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PREFACE 

This  guide  is  the  result  of  work  conducted  under  Naval  Air  Development 
Center  Contract  No.  N62269-76-C-0434  between  1 July  1976  and  30  June 
1977.  Whereas  the  objective  of  this  User's  Guide  is  to  clarify  and  ex- 
pand on  the  test  and  evaluation  sections  of  MIL-H-46855,  this  is  only  one 
phase  of  an  effort  to  develop  user's  guides  for  the  major  portions  of 
MIL-H-46855.  Phase  I,  which  was  completed  June  1976,  covered  the  analysis 
portions  of  MIL-H-46855. 

The  author  is  indebted  to  the  following  persons  for  their  guidance  and 
contributions : 

CDR  Paul  R.  Chatelier,  Naval  Air  Systems  Command 
i Dr.  Lloyd  Hitchcock,  Naval  Air  Development  Center 

; Mr.  Ed  L.  Holshouser,  Pacific  Missile  Test  Center 

LCDR  William  F.  Moroney,  Pacific  Missile  Test  Center 

Within  the  Boeing  Aerospace  Company,  the  program  was  directed  by  Mr.  W.  J. 

; Hebenstreit  of  Logistics  Support  and  Services'  Crew  Systems  Organization. 

Other  Boeing  personnel  whose  consultation  during  the  program  contributed 
! significantly  to  the  information  contained  herein  include;  Mr.  D.  W.  Jahns, 

Dr.  G.  R.  Gardlin,  and  Mr.  M.  M.  Nagata. 
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I.n  INTRODUCTION 


During  the  course  of  system  development,  test  and  evaluation  (T&E)  is  re- 
quired to  assure  that  the  evolving  system  meets  a myriad  of  requirements 
ranging  from  reduction  of  technical  risks  through  demonstrating  the  sys- 
tem's military  utility.  Generally,  two  broad  categories  of  test  and  evalu- 
ation are  used  (Reference  1);  Development  Test  and  Evaluation  (DT&E)  and 
Operational  Test  and  Evaluation  (OT&E),  where  the  former  is  planned,  con- 
ducted, and  monitored  by  the  Developing  Agency  (DA)  of  the  system,  and  the 
latter  is  the  responsibility  of  the  Operational  Test  and  Evaluation  Force 
(OPTEVFOR).  A third  category.  Production  Acceptance  Test  and  Evaluation 
(PAT&E),  is  testing  conducted  on  production  items  to  demonstrate  that  systems 
meet  contract  requirements  and  specifications. 


The  first  two  categories  of  T&E  are  defined  in  Reference  1 as  follows: 

"DT&F  is  that  test  and  evaluation  conducted  to:  demonstrate  that 
the  engineering  design  and  development  process  is  complete;  demon- 
strate that  the  design  risks  have  been  minimized;  demonstrate  that 
the  system  will  meet  specifications;  and  estimate  the  system's  mili- 
tary utility  when  introduced..." 

"OT&E  is  that  test  and  evaluation  conducted  to  estimate  the  prospec- 
tive system's  military  utility,  operational  effectiveness,  and  opera- 
tional suitability  (including  compatibil ity,  interoperability,  relia- 
bility, maintainability,  and  logistic  and  training  requirements), 
and  need  for  modifications.  In  addition,  OT&E  provides  information 
on  organization,  personnel  requirements,  doctrine,  and  tactics.  Also, 
it  may  provide  data  to  support  or  verify  material  in  operating  instruc- 
tions, publications,  and  handbooks..." 

Human  factors  engineering  (HFE)  is  an  extremely  important  aspect  of  the  total 
T&E  effort.  HFE  T&E  must  be  establistied  to: 

a)  demonstrate  conformance  of  system,  equipment  and  facility 
design  to  human  engineering  design  crtieria; 

b)  confirm  compliance  with  performance  requirements  where  man 
is  a performance  determinant; 

c)  secure  quantitative  measures  of  system  performance  which  are  a 
function  of  man-machine  interaction;  and 


^OPNAVINST  3960.10.  Department  of  the  Navy.  Office  of  the  Chief  of  Naval 
Operations,  "Test  and  Evaluation",  Washington,  D.  C.,  October  1975. 
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d)  determine  whether  undesirable  design  or  procedural  features 
have  been  introduced. 

1.1  Purpose  of  the  Guide 

The  objective  of  this  guide  is  to  provide  assistance  to  human  factors  (HF) 
engineers  in  the  planning,  scheduling,  and  performance  of  human  factors 
engineering  (HFE)  T&E  in  accordance  with  the  requirements  of  MIL-H-46855 
(Reference  2).  The  requirements  of  Paragraphs  3.2.4  through  3. 2. 4. 3 of 
MIL-H-46855  are  interpreted  ana  delineated  in  terms  of  the  HFE  T&E  steps  and 
techniques  which  need  to  be  performed.  This  guide  will  halp  to  ensure  that 
the  HF  engineer  includes  all  appropriate  aspects  of  the  total  HFE  T&E  pro- 
cess in  system  development  efforts. 

There  has  been  a long  standing  need  for  both  this  type  of  guide  and  for  a 
guide  which  provides  assistance  to  human  factors  engineering  (HFE)  analysts 
in  performing  the  analysis  portion  of  their  engineering  activities.  The 
analyst's  guide  (Reference  3)  has  been  prepared  as  the  first  phase  of  a total 
effort  to  amplify  and  interpret  the  requirements  of  MIL-H-46855. 

The  particular  need  for  this  T&E  guide  has  been  to  select  effectively,  for  a 
given  type  of  program,  just  which  HFE  T&E  techniques  should  be  performed, 
when  and  how  to  perform  them,  how  their  results  will  be  used,  and  what  their 
relative  cost  will  be.  These  technique  selection  criteria  enable  the  HF 
engineer  to  develop  realistic,  objective  planning  within  the  monetary  and 
time  constraints  of  a program.  This  document  provides  guidelines  for  the 
HF  engineer  that  are  based  on  both  Navy  and  industry  surveys  of  T&E  exper- 
ience and  needs.  The  guide  lists  and  explains  the  criteria  for  choosing 
various  T&E  techniques.  It  provides  the  basis  for  matching  particular  tech- 
niques to  particular  applications,  and  it  describes  in  detail  how  to  use  the 
various  techniques. 

MIL-H-46855A , Human  Engineering  Requirements  for  Military  Systems,  Equipment 
and  Facilities,  2 May  1972. 

^Geer , C . W . , Ana lys t_' s Gu i de  for  the  Analysis  Sections  Sections  of  MIL-H-46855, 
0180-19476-1,  Boeing  Aerospace  Company,  Naval  Air  Development 
Center,  June  1976. 
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This  guide  may  be  used  at  all  levels  of  program  design  and  procurement, 
including  analysis;  however,  it  is  designed  more  specifically  for  the  HF 
engineer  to  use  during  the  later  design  development  test  and  evaluation 
stages. 

1.2  Scope  of  the  Guide 

This  document  assumes  user  knowledge  of  basic  HFE  concepts.  It  presents 
the  significant  aspects  of  HFE  T&E  alone  rather  than  material  on  the  com- 
plete field  of  human  factors  engineering.  A complete  description  of  human 
factors  engineering  as  well  as  its  typical  applications  and  value  may  be 
found  in  any  of  several  HFE  text  books. 

This  guide  contains  a detailed  description  of  specific  HFE  T&E  tasks  and 
techniques.  It  indicates  how  to  go  about  implementing  or  using  'hese  tasks 
and  techniques  and  the  best  time  to  use  them.  Comments  are  inc.  ;ded  re- 
garding technique  selection  and  usefulness.  In  many  instances,  examples 
of  particular  techniques  are  also  presented. 

The  guide  will  be  of  use  in  responding  to  RFP's  (Request  for  Proposals) 
and  in  the  preparation  of  HFE  Test  and  Evaluation  Plans  and  HFE  Program 
Plans  in  particular.  A program  planning  decision  tree  checklist  for  HFE 
observers/evaluators  is  included  in  Section  2.3  of  this  guide.  This  de- 
cision tree  checklist  indicates  the  decisions  to  be  made  and  actions  to 
be  taken  in  responding  to  an  RFP  or  during  the  performance  of  the  MIL-H- 
46855  T&E  effort. 

A similar  guide  to  this  has  been  prepared  for  use  by  Navy  managers  (Ref- 
erence 4).  It  includes  most  of  the  same  material;  the  major  difference  is 
in  the  detail  presented  in  the  sections  on  HFE  T&E  tasks  and  techniques. 

The  section  on  program  planning  in  the  Navy  Manager's  Guide  is  modified 
appropriately  to  the  manager's  point  of  view. 


4 

Geer , C . W . , Navy  Manager's  Guide  for  the  Test  and  Evaluation  Sections 
of  MIL-H-46855,  D194-1006-2,  Boeing  Aerospace  Company,  Naval  Air  Develop- 
ment Center,  'June  1977. 
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In  Appendix  A there  is  a list  of  all  the  various  organizations  that  were 
contacted  in  order  to  determine  the  kinds  of  data  which  should  be  contained 
in  the  two  guides.  These  individuals  were  interviewed  directly  as  to  their 
T&E  experience,  problems,  and  needs. 

The  appendices  also  contain  information  on  definitions/acronyms  pertaining 
to  this  guide,  and  the  T&E  sections  of  MIL-H-46855.  Appendix  D is  a com- 
plete package  of  T&E  technique  worksheets.  These  worksheets  were  prepared 
from  data  obtained  from  survey  of  individuals  listed  in  Appendix  A and 
through  a survey  of  available  literature.  The  worksheets  were  prepared 
primarily  for  the  purpose  of  gathering  data  for  the  narrative  descriptions 
f of  each  of  the  techniques  in  Section  5.0.  However,  the  worksheets  will  be 

I of  direct  use  for  technique  comparisons. 
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2.0  HFE  TEST  AND  EVALUATION  APPROACH 


This  section  of  the  guide  describes  a general  approach  to  the  HFE  T&E 
effort.  It  includes  both  the  formal  and  the  practical  (or  technical) 
reasons  for  performing  HFE  T&E.  The  approach  to  this  HFE  T&E  effort  is 
detailed  by  a program  planning  decision  tree  diagram  and  by  a section 
describing  the  basic  HFE  T&E  technique  considerations  of  a)  data  inputs, 
b)  timing,  c)  level  of  evaluation  detail,  d)  evaluator's  judgment,  and 
e)  evaluation  applications. 

2.1  Formal  Requirements 

The  formal  requirements  for  performing  test  and  evaluation  are  found  in 
certain  Department  of  Defense  (DoD)  directives  and  SECNAV  (‘^ccieLary  of  Navy), 
NAVMAT  (Chief  of  Navy  Material),  and  OPNAV  (Chief  of  Naval  Operations)  in- 
structions. The  particular  formal  requirements  for  HFE  and  HFE  T&E,  are 
found  in  NAVMATINST  3900.9  (Reference  5).  Additional  requirements  for  HFE 
T&E  are  also  found  in  MIL-H-46855.  Two  Data  Item  Descriptions  (DID's)  have 
been  developed  for  use  in  performing  HFE  T&E.  There  are  also  several  guides, 
handbooks,  and  general  literature  sources  on  the  subject  of  HFE  T&E.  Some 
of  the  most  important  formal  HFE  T&E  requirements  are  always  to  be  found  in 
the  program  contract,  including  the  system  specification  and  statement-of- 
work.  While  the  military  is  largely  controlled  by  directives  and  instruc- 
tions, contractors  are  controlled  by  military  specifications/standards  and 
DID's.  Guides,  handbooks,  general  literature  and  the  attitudes  of  both  the 
procurement  agency  and  contractor  tie  military  and  contractor  HFE  T&E  efforts 
together. 

2.1.1  T&E  Directives  and  Instructions 

In  1971,  the  Deputy  Secretary  of  Defense  promulgated  the  policy  for  major 
defense  system  acquisition  via  DoD  Directive  5000.1,  "Major  Systems 
Acquisition"  (Reference  6).  Included  within  this  directive  is  the 


^NAVMATINST  3900.9,  Department  of  Navy,  Headquarters  Naval  Material  Command, 
"Human  Factors",  Washington,  D.  C.  20360,  September  1970. 

^DoD  Directive  5000.1,  Department  of  Defense,  "Major  System  Acquisition", 
Washington,  D.C.  20301,  January  1977. 
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requirement  to  formalize  and  incorporate  T&E  into  the  early  program  devel- 
opment stages.  This  directive  also  required  the  Navy  to  upgrade  and  ex- 
pand T&E  of  all  weapon  systems  to  include  testing  in  a simulated  combat 
environment  as  a prerequisite  for  procurement.  SECNAVINST  5000.1  imple- 
mented this  system  acquisition  policy  within  the  Navy.  Reference  7 pre- 
sents the  Defense  System  Acquisition  Review  Council  (DSARC)  review  require- 
ments for  weapons  systems  acquisition,  and  translates  the  review  checklists 
(DSARC  milestones)  into  HFE  requirements. 

DoD  Directive  5000.3,  "Test  and  Evaluation"  (Reference  8)  establishes  pol- 
icy for  the  conduct  of  test  and  evaluation  by  the  Military  Departments  and 
Defense  Agencies.  It  directs  the  developing  agency  of  the  DoD  component  to 
initiate  T&E  as  early  as  possible  and  to  continue  throughout  the  system  ac- 
quisition process  as  necessary  to  assist  in  progressively  reducing  acquisi- 
tion risks  and  in  assessing  military  worth.  In  addition,  DoD  Directive 
5000.3  requires  that  the  acquisition  schedules  be  based  upon  accomplished 
T&E  milestones  prior  to  the  time  key  decisions  are  to  be  made,  which  would 
commit  significant  additional  resources. 

NAVMATINST  3960.6,  "Planning  and  Implementation  of  Tests  and  Evaluations  of 
New  Weapon  Systems",  (Reference  9)  provides  guidance  for  the  planning  and 
implementation  of  tests  and  evaluations  required  as  a part  of  the  system 
acquisition  process.  This  instruction  provides  guidance  for  preparing  the 
Test  and  Evaluation  Master  Plan  (TEMP). 

OPNAVINST  3960.10,  "Test  and  Evaluation",  (Reference  1)  establishes  policies 
for  T&E  in  Navy  acquisition  programs  and  defines  T&E  responsibilities 


^Holshouser,  E.  L.,  Translation  of  DSARC  Milestones  into  Human  Factors  En- 
aineering  Requirements,  Pacific  Missile  Test  Center,  Point  Mugu,  Calif., 
93042,  TP-45-58,  AD-3006927L,  September  1975. 

D Directive  5000.3,  Department  of  Defense,  "Test  and  Evaluation",  Wash- 
ington, D.  C.  203-1,  January  1973. 

^NAVMATINST  3960.6,  Department  of  Navy,  Headquarters  Naval  Material  Com- 
mand, "Planning  and  Implementation  of  Tests  and  Evaluations  of  New  Weapons 
Systems",  Washington,  D.  C.  20350,  August  1973. 
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for  various  organizations.  It  establishes  procedures  for  planning,  con- 
ducting, and  reporting  T&E.  It  describes  the  relationship  between  develop- 
mental and  operational  T&E  and  it  establishes  procedures  and  a format  for 
test  and  evaluation  master  plans  (TEMP's).  It  also  establishes  procedures 
for  obtaining  fleet  RDT&E  (Research  Development  Test  and  Evaluation)  Research 
and  Development  (R&D)  that  is  not  part  of  an  acquisition  program.  Three  dis- 
tinct phases  of  T&E  are  defined  by  this  instruction.  The  relationship  be- 
tween these  phases  is  discussed  in  Section  3.0. 

NAVMATINST  3900.9,  "Human  Factors",  establishes  policies  and  requirements 
necessary  to  insure  adequate  development  of  human  factors  aspects  of  sys- 
tems and  equipment.  This  includes  all  development,  test  and  evaluation, 
and  production  programs  and  projects.  This  instruction  states  that  the 
human  element  of  the  Navy  systems  shall  undergo  the  same  development,  test 
and  evaluation  steps  as  equipment  elements  of  the  same  system. 

2.1.2  Military  Specifications  and  Standards 

The  primary  source  for  contractor  HFE  T&E  requirements  is  MIL-H-46855.  A 
copy  of  the  sections  pertinent  to  analysis  is  contained  in  Appendix  C.  Fig- 
ure 2.1-1  shows  each  of  the  major  MIL-H-46855  sections  in  functional  rela- 
tion to  each  other.  Section  3.2.4  "Perform  HE  Test  and  Evaluation"  is  the 
function/section  most  pertinent  to  this  guide.  It  is  divided  into  three 
subsections:  planning,  implementation,  and  failure  analysis.  Unless  other- 
wise contractually  noted.  Section  3.2.4  and  its  subsections  are  the  detailed 
requirements  in  MIL-H-46855  necessary  to  provide  an  adequate  HFE  T&E  program. 
The  major  paragraph,  3.2.4,  indicates  most  of  the  tasks  that  the  contractors 
must  perform.  Paragraph  3. 2. 4.1  (Planning)  emphasizes  the  need  for  HFE  tests 
integrated  along  with  other  system  tests  in  order  to  save  the  cost/duplica- 
tion of  completely  separate  testing.  Early  testing,  in  accordance  with  this 
paragraph  will  also  save  the  cost  of  unnecessary  late  system  rework.  Para- 
graph 3. 2. 4. 2 (Implementation)  contains  a list  of  test  tasks  to  be  performed 
in  order  to  insure  a complete  HFE  T&E  program.  Paragraph  3. 2. 4. 3 (Failure 
Analysis)  emphasizes  the  need  to  investigate  human  error.  HFE  T&E  techniques 
listed  in  later  sections  of  this  guide  may  be  used  to  accomplish  this  require- 
ment for  human  failure  analysis. 
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• The  Primary  source  of  HFE  design  criteria  to  test  to  is  in  MIL-STD-1472 

[ (Reference  10).  This  standard  presents  HE  design  criteria,  principles, 

I and  practices  to  be  applied  in  the  design  of  systems,  equipment  and  facili- 

ties. The  purpose  of  the  HFE  T&E  in  regard  to  this  standard  is  to  insure; 
operator/maintainer  required  performance  is  achieved;  skill,  personnel-equip- 
ment combinations  is  achieved;  and  the  design  within  and  among  systems  is 
standardized.  There  are  several  other  standards  that  contain  design  cri- 
teria which  may  need  to  be  verified  during  HFE  T&E.  However,  they  are  too 
numerous  to  list  all  of  them  herein.  MIL-L-25467  (Instrument  Lighting), 
MIL-STD-411  (Air  Crew  Station  Signals),  and  MIL-STD-1333  (Air  Crew  Station 
Geometry)  are  examples  of  other  standards  that  contain  important  HFE  design 
criteria. 

In  addition  to  MIL-H-46855,  the  following  specifications  are  of  some  impor- 
tance to  the  HFE  T&E  effort:  MIL-D-8706  (Engineering  Data  and  Test  Contract 
Requirements  for  Aircraft  Weapon  Systems);  MIL-D-8708  (Demonstration  Require- 
ments for  Airplanes) ; MIL-D-23222  (Demonstration  Requirements  for  Helicopters); 
MIL-M-8650  (General  Specification  for  Aircraft  Mockups);  and  MIL-M-18828 
(Construction  of  Mockups  for  Guided  Missiles). 

2.1.3  Data  Item  Descriptions 

If  called  for  in  the  system  program  contract,  the  Data  Item  Descriptions 
(DID's)  are  extremely  important  to  the  HFE  T&E  effort  as  performed  by  the 
contractor.  The  two  most  important  DID's  pertaining  to  HFE  T&E  are  DI-H- 
2105,  Human  Engineering  Test  Plan  and  DI-H-2111,  Human  Engineering  Test  Re- 
port. The  HE  Test  Plan  DID  describes  in  detail  how  to  prepare  the  contract- 
or's test  plan.  It  describes  the  proposed  approach  taken  for  obtaining  T&E 
data.  It  establishes  and  explains  all  standards,  tests,  and  associated 
analyses.  It  also  establishes  other  means  that  will  constitute  adequate 
proof  upon  completion  of  the  development  phase  that  acceptable  levels  of 


^®MIL-STD-1472B,  Human  Engineering  Design  Criteria  for  Military  Systems, 
Equipment  and  Fac'ilities,  31  December  1974. 
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human  performance,  time,  accuracy,  and  safety  factors  can  be  achieved  in 
operational  use  under  specified  manning  levels.  Reference  11  is  particu- 
larly helpful  in  preparing  the  HF  Test  Plan.  The  HE  Test  Report  DID  des- 
cribes how  to  prepare  the  documentation  associated  with  contractor  tests. 

The  test  report  is  to  be  used  by  the  procuring  activity  to  assure  that  the 
man-equipment  interface  requirements  for  the  operation  and  maintenance  of 
the  system  conform  to  the  contractual  requirements. 

2.1.4  Guides,  Handbooks,  and  General  Literature 

This  category  of  HFE  T&E  reouirements  is  by  far  the  most  voluminous,  and 
it  includes  several  data  sources  of  note.  The  Department  of  the  Navy  RDT&E 
Management  Guide  (Reference  12)  is  a good  source  of  general  Integrated  Logis- 
tics Support  (ILS)  T&E  requirements.  Reference  13  is  an  excellent  summary 
of  policy  and  procedures  for  HFE  T&E  which  can  serve  as  a useful  guide  for 
gaining  an  overview  of  the  scope  of  the  HFE  T&E  effort.  Reference  14,  Hu- 
man Factors  Evaluation  in  System  Development,  is  an  excellent  textbook  source 
for  HFE  T&E  data.  General  theory  as  well  as  data  collection  and  analysis 
methods  are  thoroughly  presented. 

The  Air  Force  Human  Resources  Laboratory  has  prepared  a report  (Reference  15) 
which  reviewed  95  documents  related  to  Personnel  Subsystem  Test  and  Evalua- 
tion (PSTE).  Each  report  was  abstracted  and  then  analyzed  in  terms  of  11 
categories:  a)  Scope  and  relation  to  HF  elements;  b)  Test  objectives;  C)  Data 
requirements  and  test  criteria;  d)  Data  collection  methodology;  e)  Support 


Holshouser,  E.  L.,  Guide  to  Human  Factors  Engineering  General  Purpose 
Test  Planning  (GPTP),  Pacific  Missile  Test  Center,  Point  Mugu,  Calif., 
93042,  1st  Iteration,  September  1976. 

^^Oepartment  of  the  Navy  Management  Guide,  NAVSO  P-2457  (Rev.  1-75),  1 
January  1975. 

13 

Holshouser,  E.  L.,  Human  Factors  Engineering  Policy  and  Procedures  for 
Test  and  Evaluation  of  Navy  Systems,  TP-75-15,  PacVic  Missi  iV'Test 
Center,' A'D-B006035L,  July  l'975. 

14 

Meister,  D. , and  Rabideau,  G.  F.,  Human  Factors  Evaluation  in  System 
Development,  John  Wiley  and  Sons,  NewTork,  1965. 

1 5 

Askren,  W.  B.,  and  Newton,  R.  R.,  Review  and  Analysis  of  PersonnelSub- 
system  Test  and  Evaluation  Literature,  7TFHRL-TR-i^-7,  TfFSC,  W-MFB, 

January  1969. 
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requirements;  f)  Reducing  and  analyzing  data;  g)  Significant  test  results; 
h)  Communicating  and  using  test  results;  i)  Factors  in  planning  a test  pro- 
gram; j)  Factors  in  conducting  a test  program;  and  k)  Other  problems.  In- 
cluded among  the  several  excellent  references  within  the  AFHRL  report  are 
three  particularly  useful  documents: 

a)  The  first  report  (Reference  16)  is  prepared  by  the  Aerospace 
Medical  Research  Laboratories.  It  describes  the  practices 
and  evaluative  aspects  of  human  performance  assessment  in  Air 
Force  systems.  The  human  performance  test  program  for  34  sys- 
tems and  subsystems  representing  the  major  types  of  systems  used 
by  the  Air  Force  were  reviewed. 

b)  The  Reference  17  guidebook  was  developed  as  an  aid  for  the 
Project  Officer  in  the  assessment  of  human  factors  effects 

on  system  performance  and  in  the  isolation  of  causal  factors. 
Included  in  this  guide  are  the  tools  and  techniques  for  system 
evaluation.  This  includes  methods  for  obtaining  time,  accuracy, 
and  maintenance  data  as  well  as  the  techniques  for  analyzing  and 
interpreting  these  data.  A detailed  example  of  the  application 
of  the  techniques  is  included. 

c)  Reference  18  is  a good  review  of  the  problems  associated  with 
field  testing  of  human  performance  and  the  resulting  subjectively- 
oriented  data  collection  techniques.  Factors  to  be  consid- 
ered in  the  planning  of  field  tests  were  also  discussed. 


^Keenan,  J.  J.,  et.  al..  Concepts  and  Practices  in  the  Assessment  of  Human 
Performance  in  Air  Force  Systems,  AMRL-TR-168,  AD  625  "041 ,'  Aerospace 
Medical  Research  laboratories,  W-PAFB,  September  1965. 

^^Myers,  L.  B.,  et.  al..  Guidebook  for  the  Collection  of  Human  Factors  Data. 
Report  PTB  66-3,  AD  631  023,  HRB  - Singer,  Inc.,  State  College,  Pennsyl- 
vania, January  1966. 

^*^Rabideau,  G.  F.,  "Field  Measurement  of  Human  Performance  in  Man-Machine 
Systems",  Humaji  Factors , December  1964. 
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2.2  Technical  Requirements 

These  are  the  important  "real  world"  requirements  such  as  design  criteria, 
safety,  training,  and  life  support  that  the  HFE  observer/evaluator  needs 
to  ensure  are  verified  as  integral  to  the  system  design  being  evaluated. 

The  system  must  meet  these  requirements  in  order  to  insure  the  adequacy 
of  the  man/machine  interface.  Table  2.2-1  lists  each  of  these  require- 
ments in  a matrix  comparison  with  the  techniques  presented  in  Section  5.0. 

The  purpose  of  this  matrix  is  to  quickly  show  which  techniques  may  be  used 
to  test  and  evaluate  the  system  in  order  to  verify  compliance  with  these 
technical  requirements.  Each  of  these  technical  requirements  is  presented 
in  the  following  subsections  along  with  a brief  sunmary  of  the  reasons  for 
the  importance  of  their  test  and  evaluation. 

2.2.1  Human  Performance 

All  systems  require  a degree  of  human  performance  in  order  to  function  as 
specified.  In  order  to  meet  system  performance  requirements  such  as  speed, 
maneuverability,  range  or  turnaround  time,  the  operators  and  maintainers 
must  meet  certain  minimum  requirements  for  performing  their  assigned  tasks. 
These  requirements  are  most  always  in  terms  of  time  to  perform  a task  or 
accuracy  (or  reliability)  with  which  a task  must  be  performed.  The  planned 
opera  tor/ maintainer  task  times  and  error  rates  must  therefore  be  verified 
by  HFE  observation  and  test. 

2.2.2  Design  Criteria 

Human  engineering  design  criteria  are  the  kinds  of  requirements  contained  in 
MIL-STD-1472  (Reference  Section  2.1.2).  The  incorporation  of  these  criteria 
into  the  hardware  design  insures  a relatively  high  degree  of  opera tor/main- 
tainer  performance.  These  criteria  are  based  on  several  years  of  experience 
as  to  the  design  features  or  details  that  tend  to  minimize  operator  errors 
or  slow  operator  performance.  Many  of  these  design  requirements  are  based 
on  lab  experiments  and  quantified  comparative  data.  For  each  of  the  basic 
system  functions,  there  should  be  certain  HE  criteria  that  should  be  incor- 
porated into  the  design  and  should  therefore  be  evaluated  to  insure  its  exis- 
tence. General  categories  of  HE  design  criteria  include:  displays,  controls, 
labeling,  workspace  layout,  and  maintainability. 
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Table  2.2-1:  HFE  T&E  Technical  Requirements  vi.  Techniques 
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DESIGN  CRITERIA  CHECKLIST 
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SOUND  TAPES 
VIDEO  TAPES 


PHOTOGRAPHY 
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AUTOMATIC  PHYSIOLOGICAL; 

PHYSIOLOGICAL  INSTRUMENTAITON 
PHYSICAL  MEASUREMENT 
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2.2.3  Safety 

This  is  a special  category  of  design  criteria  that  is  listed  here  and  in 
other  documentation  separately  in  order  to  add  emphasis  to  its  importance. 

As  is  the  case  with  other  design  criteria,  lab  tests  and  experience  have 
indicated  that  certain  design  requirements  (e.g.,  rails  on  stairs)  inher- 
ently increase  the  system  safety. 

2.2.4  Training 

Deoending  on  the  organizational  setup  or  "charter"  of  a particular  HFE  T&E 
group,  the  evaluation  of  test  participant  training  may  or  may  not  be  neces- 
sary by  the  HFE  group.  Certainly  training  is  an  important  aspect  of  a total 
system  test.  All  other  HFE  aspects  may  be  properly  provided  for  but  im- 
proper training  of  participants  may  be  sufficient  cause  to  ruin  an  otherwise 
successful  test.  Training  requirements  may  be  evaluated  in  tertns  of  type, 
duration , and  detai 1 . 

2.2.5  Personnel  Ski  11 /Quantity 

Closely  associated  with  training  is  the  test  participant  skill  level.  This 
is  a combination  of  his  training  on  other  systems  somewhat  related  to  the 
present  system  being  tested  and  his  aptitude  for  the  type  of  work  he  is  per- 
forming. Quantity  simply  refers  to  the  number  of  test  participants  perform- 
ing each  of  the  operator/maintainer  tasks.  The  fact  that  the  proper  number, 
too  many,  or  too  few  operators  are  assigned  to  a test  task  should  be  ob- 
served and  reported. 

2.2.6  Technical  Publi cations 

This  technical  requirement  is  similar  to  the  training  requirements  in  that 
it  is  often  not  the  job  of  HFE  observers  to  evaluate.  However,  technical 
publications  (tech  orders  or  job  manuals)  are  a very  important  part  of  the 
total  system  under  test.  In  most  instances,  test  participants  operate  or 
maintain  the  system  under  test  in  accordance  wi th  procedures  that  have  been 
developed  for  the  particular  system  function.  Just  as  it  is  necessary  that 
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the  prototype  hardware  be  tested  and  evaluated,  so  must  the  "prototype" 
or  preliminary  technical  publications  be  tested.  This  evaluation,  which 
is  sometimes  referred  to  as  validation/verification,  should  include  how 
well  the  technical  data  meets  the  requirement  to  provide  easily  under- 
standable and  accurate  procedural  data.  Technical  narrative,  diagrams, 
illustrations,  and  photographs  should  all  be  evaluated. 

2.2.7  U fe  Support  Criteria 

Life  support  criteria  is  another  special  category  of  design  criteria.  It 
is  listed  here  separately  from  design  criteria  to  add  emphasis  to  these 
requirements.  Life  support  requirements  include:  atmospheric  conditions, 
noise,  vibration,  shock,  toxicology,  radiology,  lighting,  psychophysiology, 
fatigue,  clothing,  and  personal  equipment.  In  many  man/machine  system  de- 
signs, life  support  requirements  are  relatively  insignificant.  However, 
in  high  performance  systems  and  those  requiring  closed  loop  environmental 
control,  life  support  requirements  are  particularly  critical. 

2.3  Decision  Making  Structure 

There  are  a considerable  number  of  decisions  that  must  be  made  and  actions 
taken  by  the  HFE  observer/evaluator  during  the  HFE  T&E  program  effort  phase. 
The  decision  tree  fold-out  provided  in  this  section  of  the  guide  (Figure 
2.3-1)  is  designed  to  aid  the  HFE  observer/evaluator  in  determining  the 
proper  decisions  and  actions.  Its  further  purpose  is  to  provide  the  HFE 
observer/evaluator  with  planning  assistance  and  checklist  items  to  ensure 
inclusion  of  all  necessary  tasks  and  data  in  the  T&E  program.  This  diagram 
is  designed  to  be  an  adjunct  to  the  General  Purpose  Test  Planning  (GPTP) 
(Reference  11)  diagrams. 

The  Human  Engineering  Test  Plan  is  developed  as  a result  of  specified  pro- 
gram office  requirements  and  Data  Item  Description  (DID's)  which  are  called 
out  in  program  statements-of-work  (SOW's).  The  GPTP  indicates  how  to  pre- 
pare this  Human  Engineering  Test  Plan  which  is  the  documentation  of  how  the 
HFE  effort  will  be  accomplished.  The  GPTP  diagrams  are  designed  to  assist 
the  HFE  observer/evaluator  in  preparing  the  HE  test  plan.  The  diagram  in 
this  section  is  intended  to  assist  in  all  aspects  of  HFE  T&E,  particularly 
the  initial  phase  of  setting  up  an  HFE  T&E  effort. 
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The  pertinent  decisions  and  possible  actions  have  been  placed  on  the  flow 
chart  diagram  in  blocks  adjacent  to  other  associated  or  related  decision 
and  action  blocks.  Arrows  are  provided  between  the  blocks  to  indicate  the 
direction  of  sequence  of  task  action.  The  placement  of  several  blocks  on 
a single  page  with  connecting  arrows  allows  the  HF  engineer  to  quickly  see 
relationships  between  each  of  the  tasks  that  he  must  accomplish.  A narra- 
tive description  of  these  same  HFE  T&E  tasks  and  task  relationships  would 
undoubtedly  be  hopelessly  complex. 

2.4  Basic  Considerations 

There  are  several  important  considerations  pertaining  to  the  HFE  T&E  effort 
which  are  in  addition  to  the  approp-^iate  sections  of  MIL-H-46855,  the 
program  manager's  guidelines,  the  company  or  organization  policy,  and 
the  other  constraints  indicated  in  Sections  2.1,  2.2,  and  2.3.  These 
considerations  consist  of  the  type  of  data  required  to  start  an  HFE  T&E 
effort,  the  timing  as  to  when  to  perform  the  T&E,  the  level  of  T&E  detail 
required,  the  use  of  the  HFE  evaluator's  judgment,  and  the  type  of  specific 
results  normally  expected  from  the  HFE  T&E  effort.  Section  5.0,  "T&E  Tech- 
niques", deals  with  these  basic  considerations  in  relation  to  specific  HFE 
T&E  techniques,  but  this  section  pertains  to  these  basic  considerations  in 
relation  to  the  overall  HFE  T&E  effort. 

2.4.1  Data  Inputs 

These  consist  of  criteria  or  human  performance  requirements  that  are  to 
be  determined  or  verified  by  T&E.  They  may  be  system  or  mission  require- 
ments established  early  in  the  program.  They  may  be  military  specifi- 
cations and  standards  criteria.  Operator/maintainer  requirements  (and 
assumptions)  developed  out  of  the  MIL-H-46855  analysis  effort  should  be 
available  for  evaluation.  Ideally,  these  data  are  available  from  a pro- 
gram data  retrieval  system  (e.g.,  FLAG).  They  may  be  available  from 
another  technology  than  HFE.  In  any  case,  T&E  should  be  conducted  against 
set  test  criteria  (success  criteria)  and  if  the  data  does  not  exist,  it 
must  be  generated.  Also,  the  TEMP  (Reference  1)  and  HFTEMAN  (Reference 
Section  5.1.7)  should  be  used  as  inputs. 
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During  early  program  phases,  all  of  these  data  are  tested  against  drawings 
and  static  mockups.  As  the  program  progresses  and  T&E  becomes  more  formal, 
more  sophisticated  mockups  and  prototype  hardware  are  used. 

2.4.2  Timing 

Without  the  proper  scheduling  of  the  HFE  T&E  effort,  it  can  turn  out  to  be 
of  little  use  to  the  system  design.  It  is  not  sufficient  just  to  perform 
HFE  T&E.  It  is  equally  important  to  demonstrate  that  the  results  of  the 
effort  will  be  completed  or  partially  completed  at  a point  in  the  schedule 
when  it  can  properly  impact  the  system  design.  All  too  often,  HFE  T&E  is 
performed  as  an  after-the-fact  documentation  exercise  or  just  a workaround 
procedure  that  appears  in  a technical  publication.  The  later  the  design 
is  tested,  the  less  chance  there  is  to  modify  the  crew  station  or  other 
man/machine  interface.  Late  findings  of  serious  crew  system  problems  can 
be  extremely  expensive,  both  in  accidents  and  system  redesign. 

Appendix  D shows  the  stage  or  phase  of  a program  for  which  a particular 
technique  is  best  suited. 

2.4.3  Level  of  Detail 

The  level  of  T&E  detail  that  must  be  performed  is  significant  to  the  HFE 
manpower  effort.  If  the  wrong  emphasis  is  given  to  the  amount  of  detailed 
data  gathering,  HFE  T&E  may  be  wasted  by  either  obtaining  data  that  has  no 
use  or  by  failing  to  obtain  sufficiently  detailed  data  to  be  useful.  It  is 
the  job  of  the  HFE  observer  or  manager  to  decide  what  level  of  T&E  will  lead 
to  worthwhile  data  or  useful  design  criteria.  For  example,  new  system  de- 
signs or  programs  often  contain  requirements  that  are  identical  to  previous- 
ly designed  and  tested  systems.  There  is  probably  no  point  in  repeating  a 
detailed  HFE  T&E  of  all  the  system  functions  and  operator  tasks  that  have 
already  been  evaluated.  It  is  simply  not  cost  effective,  especially  when 
new  program  schedules  and  manpower  budgets  generally  are  extremely  limited. 
The  old  system  test  and  evaluation  results  should  provide  all  the  data  that 
would  be  needed  for  that  portion  of  the  new  system  design  criteria. 
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The  effort  that  should  be  expended  during  the  new  program  is  on  the  new  and 
potentially  critical  functions  and  tasks.  These  are  the  kinds  of  tasks  that 
appear  to  be  marginal  in  terms  of  operator  reliability,  time  performance,  or 
safety  hazards.  The  level  of  test  and  evaluation  detail  attempted  for  such 
tasks  should  be  as  far  down  as  possible  to  the  subtask  element  (indicator 
moni toring/button  pushing)  based  on  program  timing/scheduling. 

On  new  system  designs  it  may  be  necessary  to  analyze  data  down  to  as  much 
detail  as  a tenth  of  a second.  If  the  HFE  program  has  been  properly  man- 
aged, all  system  potentially  critical  tasks  will  have  been  previously  indi- 
cated for  special  HFE  T&E  considerations.  In  any  case,  the  need  to  gather 
human  performance  related  T&E  field  data  beyond  a tenth  of  a second  is  ex- 
tremely doubtful.  In  a similar  manner,  the  HFE  should  maintain  adherence 
to  the  rules  for  significant  figures  and  common  sense  when  gathering  data 
on  light  levels,  sound  levels,  reach  envelope  measurements,  etc. 

2.4.4  Evaluator's  Judgment 

The  use  of  an  experienced  HFE  observer/evaluator's  judgment  is  an  extremely 
important  basic  consideration.  The  evaluator's  judgment  or  "gut  feel"  of 
a test  situation  might  be  one  of  the  most  important  HFE  T&E  techniques  if 
it  could  be  described  more  analytically  as  a step  by  step  procedure  to 
follow.  However,  it  is  not  that  sort  of  thing.  This  consideration  is  sim- 
ply the  summary  of  the  evaluator's  significant  impressions  as  a result  of 
observing  the  system  test.  It  would  be  convenient  to  have  all  test  data 
gathered  in  an  easily  useable  quantifiable  form.  But  there  is  no  way  to 
measure  all  test  performance  parameters.  The  written  (or  spoken)  impressions 
of  the  experienced  HFE  evaluators  generally  prove  to  be  extremely  valuable 
and  cost  effective. 

Examples  of  test  information  that  are  best  described  based  on  an  experienced 
evaluator's  judgment  are:  a)  test  precision  (or  lack  of  same),  b)  workspace 
layout  adequacy,  c)  test  participant  skill,  and  d)  test  participant  train- 
ing. It  is  difficult  to  measure  these  qualities  in  any  other  way  during  a 
test.  The  results  of  the  HFE  evaluator's  observation  and  judgment  should 
be  documented  as  a part  of  the  system  test  report.  This  would  include  the 
evaluator's  "judgment"  as  to  a test  problem  solution. 


2.4.5  Appi ications 

The  end  products  of  the  HFE  T&E  effort  are  to  verify  system  design,  discover 
system  inadequacies,  provide  recommendations  for  design  or  other  system 
changes,  and  provide  information  for  a data  bank  of  human  performance  and 
crew  systems  design  related  data  to  be  used  on  later  programs.  These  out- 
puts may  be  left  in  the  form  of  the  T&E  technique  worksheets.  More  prob- 
ably, the  outputs  should  be  condensed  and  otherwise  modified  to  make  them 
more  easily  understood  by  the  program  design  personnel  and  others  who  have 
use  of  them.  (See  Reference  11  for  GPTP  guide  for  standardized  foramts). 
Table  2.4-1  shows  the  applications  for  data  developed  from  using  the  var- 
ious listed  T&E  techniques. 

It  may  be  of  use  to  HFE's  to  prioritize  T&E  results  in  some  manner  in  order 
to  show  that  there  are  certain  absolutely  essential  system  modifications 
required.  The  risk  of  not  doing  this  is  to  have  insignificant  results  acted 
upon  and  critical  data  ignored.  All  findings  must  be  well  documented  and 
files  must  be  maintained.  By  themselves,  verbal  inputs  as  to  T&E  results 
have  virtually  no  chance  of  acceptance. 
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3.0  TYPES  OF  HFE  TEST  AND  EVALUATION 

As  indicated  in  the  introductory  section  of  this  guide,  there  are  different 
types  of  T&E.  The  definitions  provided  there  are  precise  since  they  are 
based  on  the  Reference  1 instruction.  There  are  additional  detailed  defini- 
tions provided  in  Reference  1 that  are  included  in  this  section  of  the  guide. 
There  are  also  included  herein  the  less  formal  definitions  which  are  based 
on  either  system  program  schedules  or  funding  sources. 

3.1  Informal  vs.  Formal  T&E 

In  attempting  to  develop  an  integrated  HFE  T&E  technology,  one  needs  to 
recognize  that  different  types  of  test  and  evaluation  are  involved  in 
systems  development  (both  in  DT&E  ^nd  OT&E):  One,  called  HFE  "t&e" 

(lower  case)  is  predominant  in  design  concept  feasibility  efforts  and  in 
selection  from  design  configuration  alternatives.  Design  concept  feasibility 
efforts  are  exploratory  in  nature.  They  are  extremely  dependent  upon  paper 
analysis.  The  observation,  experimentation  and  testing  activities  in  this 
category  are  oriented  to  the  goal  of  developing  new  knowledge.  The  selection 
of  design  configuration  alternatives  is  a resolution  process  in  which  trade 
studies  are  performed  and  evaluated.  The  other  type  of  test  and  evaluation, 
labeled  HFE  "T&E",  is  predominant  in  system  performance  verification.  Sys- 
tem performance  verification  is  performed  with  mockups  and  prototype  hard- 
ware. The  objective  which  characterizes  this  category  is  one  of  determin- 
ing whether  a man-machine  system,  as  designed,  meets  its  performance  require- 
ments. Systems  performance  verification  is  a much  more  formal  test  and  eval- 
uation than  the  lower  case  "t&e".  It  is  this  latter  type  of  T&E  (capital 
letters)  which  is  covered  by  this  guide  since  the  applicable  paragraph  in 
MIL-H-46855  carries  the  heading  "Human  Engineering  iji  Test  and  Evaluation". 

At  the  same  time,  HFE  "t&e"  may  be  called  for  (both  exploratory  and  resolu- 
tion) when  system  program  Engineering  Change  Proposals  (ECP's)  are  required. 
The  relationship  between  these  differential  aspects  is  illustrated  in  Figure 
3.1-1. 

3.2  OPNAVINST  3960. 10  Definition 

OPNAVINST  3960.10  (Reference  1)  makes  a formal  distinction  between  three 
types  or  phases  of  T&E.  These  are  Development  Test  and  Evaluation  (DT&E), 
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Figure  3.1-1 : Informal  vs.  Formal  Aspects  of  HFE  T&E 


Operational  Test  and  Evaluation  (OT&E),  and  Production  Acceptance  Test 
and  Evaluation  (PAT&E).  These  types  of  T&E  are  shown  in  relation  to 
each  other,  to  their  subphase  elements,  and  to  the  program  phases  and 
milestones  in  Figure  3.2-1.  It  is  interesting  to  compare  this  figure 
with  the  Figure  3.1-1  definition  of  T&E  by  phases. 

3.2.1  DT&E 

DT&E  is  required  for  all  acquisition  programs  and  is  conducted  in  four 
major  phases: 

a)  DT-I  is  DT&E  conducted  during  the  conceptual  phase  to  support 
the  program  initiation  decision.  It  consists  primarily  of 
analysis  and  studies  to  derive  the  human  factors/system  require- 
ments. 

b)  DT-I I is  DT&E  conducted  during  the  validation  phase  to  support 
the  full  scale  development  decision.  It  demonstrates  the  design 
risks  have  been  identified  and  minimized.  It  consists  of  veri- 
fying the  results  of  the  special  analysis  and  studies  including 
modeling  and  simulation  on  the  critical  areas  identified  earlier. 
It  is  normally  conducted  at  the  subsystem/component  level,  up  to 
and  including  employment  of  engineering  models  for  final  evalua- 
tion. 

c)  DT- 1 1 1 is  DT&E  conducted  during  the  full  scale  development  phase 
to  support  the  first  major  production  decision.  It  demonstrates 
that  the  design  meets  its  specifications  in  performance,  relia- 
bility, maintainability,  supportabil ity,  survivability,  system 
safety,  and  electromagnetic  vulnerability.  This  phase  may  be 
further  subdivided  into  additional  phases,  such  as  Contractor 
Technical  Evaluation  (CTE)  and  formal  Navy  Technical  Evaluation 
(NTEs).  The  final  subphase  of  DT-III  is  TECHEVAL,  the  purpose  of 
which  is  to  certify  that  the  design  meets  specified  requirements 
and  is  ready  for  OPEVAL  (operational  evaluation). 
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d)  DT-IV  is  DT&E  conducted  after  the  first  major  production  decision 
to  verify  that  product  improvements,  or  correction  of  design  de- 
ficiencies discovered  during  OPEVAL,  FOT&E  (follow-on  test  and 
evaluation),  or  fleet  employment,  are  effective. 

3.2.2  OT&E 

OT&E  is  required  for  all  acquisition  programs  except  for  those  programs 
designated  by  Chief  of  Navy  Material.  OT&E  is  subdivided  into  two  major 
categories:  initial  OT&E  (lOT&E),  which  is  all  OT&E  accomplished  prior  to 
the  first  major  production  decision;  and  follow-on  OT&E  (FOT&E),  which  is 
all  OT&E  after  the  first  major  production  decision.  OT&E  is  further  divided 
into  five  major  phases  (3  lOT&E,  and  2 FOT&E): 

a)  OT-1  is  any  lOT&E  that  may  be  conducted  during  the  conceptual 
phase  to  support  the  program  initiation  decision.  Most  acquisi- 
tion programs  do  not  require  OT-I.  However,  when  an  OT-I  is  con- 
ducted, existing  systems  or  modifications  thereto  will  normally  be 
used  to  help  estimate  the  military  utility  of  the  proposed  new  sys- 
tem. 

b)  OT-I I is  lOT&E  conducted  during  the  validation  phase  to  support 
the  full-scale  development  decision.  It  provides  an  early  esti- 
mate of  projected  operational  effectiveness  and  operational  suita- 
bility of  the  system;  initiates  tactics  development;  estimates  pro- 
gram progress,  and  identifies  operational  issues  for  OT-III. 

c)  OT-III  is  lOT&E  conducted  during  the  full-scale  development  phase 
to  support  the  first  major  productiovn  decision.  OPEVAL  (Opera- 
tional evaluation)  is  the  final  subphase  of  the  OT-III.  It  con- 
sists of  a demonstration  of  achievement  of  program  objectives  for 
operational  effectiveness  and  operational  sui tabi 1 i ty , and  continu- 
ing tactics  development.  OPEVAL  normally  uses  pilot  production 
hardware,  and  begins  about  one  month  after  completion  of  TECHEVAL 
(technical  evaluation)  testing. 


31 


d)  OT-IV  is  FOT&E  conducted  after  the  first  major  production  decision, 
but  before  production  systems  are  available  for  testing.  Normally, 
OT-IV  is  conducted  with  the  same  pre-production  prototype  or  pilot 
production  systems  used  in  OPEVAL.  OT-IV  consists  of  testing  of 
fixes  to  be  incorporated  in  production  systems,  completion  of  any 
deferred  or  incomplete  lOT&E,  and  continuing  tactics  development. 

e)  OT-V  is  FOT&E  conducted  on  production  systems  as  soon  as  they  are 
available.  OT-V  provides  for  a demonstration  of  the  achievement  of 
program  objectives  for  production  system  operational  effectiveness 
and  operational  suitability.  In  addition,  OT-V  includes  OT&E  of  the 
system  in  new  environments,  or  in  new  applications,  or  against  new 
threats. 

3.2.3  PAT&E 

As  previously  indicated,  PAT&E  is  testing  conducted  on  production  items  to 
demonstrate  that  systems  meet  contract  requirements  and  specifications.  There 
are  no  subphases  to  PAT&E. 

3.3  General  Definitions 

Further  definition  of  HFE  T&E  may  be  made  from  the  standpoint  of  the 
money  source  to  pay  for  the  T&E  job,  e.g.,  if  design  engineering  money 
pays  for  an  evaluation  of  some  item  of  hardware,  then  the  evaluation  pro- 
cess is  considered  "design".  In  a similar  manner,  the  program  schedule 
helps  define  the  terms,  e.g.,  a paper  analysis  of  a system  concept  may  be 
considered  as  an  on-paper  "evaluation".  However,  since  it  may  occur  early 
in  the  program  during  the  concept  definition  phase,  it  will  most  probably 
be  considered  "analysis"  at  that  time  period. 

The  total  spectrum  of  how  to  categorize  and  differentiate  among  the  divergent 
objectives,  criteria  and  applications  of  test  and  evaluation  need  to  have  an 
agreed  upon  definition  between  customer  and  contractor  and  between  customer 
agencies  before  significant  HFE  T&E  program  work  may  occur.  A good  starting 
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point  for  establishing  a baseline  for  this  would  be  Meister's  Human  Factors: 
Theory  and  Practice  (Reference  19),  and  the  somewhat  earlier  Meister  & Rabideau 
Human  Factors  Evaluation  in  System  Development  (Reference  14). 


^^eister,  D. , Human  Factors:  Theory  and  Practice,  Wiley,  New  York,  1971. 
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4.0  T&E  METHODS 


Numerous  HFE  T&E  techniques  have  been  developed  in  response  to  different 
needs  or  characteristics  that  have  been  required  of  each.  It  is  the  intent 
of  this  section  of  the  guide  to  describe  both  the  basic  HFE  T&E  methods,  or 
general  techniques,  and  the  characteristics  that  are  used  to  evaluate  them. 
The  methods  are  described  and  evaluated  in  general  in  relation  to  the  dif- 
ferent qualities  or  characteristics  that  each  typifies.  Also,  the  use  of 
mockups,  models,  and  prototype  hardware  are  discussed  briefly  in  this  sec- 
tion. 

4.1  Technique  Evaluation  Characteristics 

The  choice  of  the  most  efficient  technique  for  performing  an  HFE  T&E  is 
a critical  step  in  the  total  evaluation  process.  The  general  technique 
or  method  chosen  will  influence  the  entire  conduct  of  HFE  T&E.  Several 
technique  characteristics  have  been  selected  for  use  in  evaluation  of  the 
different  T&E  methods/techniques.  The  techniques  may  be  compared  to  each 
other  by  use  of  narrative  comments  for  each  evaluation  characteristic  or 
by  use  of  the  Appendix  D technique  evaluation  worksheets.  The  evaluation  . 
factors  used  to  choose  the  techniques  are  described  in  the  following  sec- 
tions. 

4.1.1  Program  Use 

One  of  the  several  ways  a technique  might  be  selected  is  on  the  basis  of 
what  previous  systems/programs  have  successfully  used  that  method.  The 
HFE  evaluator  may  be  able  to  obtain  details  as  to  the  use  of  the  technique 
by  contacting  the  HFE  personnel  who  have  worked  the  previous  program.  An- 
other usage  factor  is  the  applicable  stage  or  phase  of  a program  that  the 
technique  is  best  suited  for.  Some  techniques  are  best  used  for  future 
programs  such  as  the  F-18,  or  existing  programs  such  as  the  P-3.  For  exam- 
ple, some  of  the  off-line  cockoit  Geometry  computer  programs  should  not 
be  used  in  evaluating  an  existing  P-3  cockpit.  If  the  evaluation  is  still 
needed,  it  is  easier  to  examine  the  actual  aircraft  rather  than  simulate 
it. 
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4.1.2  Inherent  Features 

Rather  than  include  all  the  inherent  characteristics  of  a technique  in  an 
applicdble/not  applicable  categorization,  these  characteristics  are  best 
considered  as  a separate  list  for  each  technique  of  all  of  the  advantages 
and  disadvantages  of  that  particular  technique.  The  inherent  validity  or 
repeatability  of  the  technique  should  be  considered  along  with  the  advan- 
tages and  disadvantages.  The  implications  of  not  using  the  technique  should 
also  be  included.  For  example,  this  could  be  a list  of  other  techniques 
which  might  yield  similar  data,  or  the  fact  that  the  technique  is  the  only 
one  available  to  provide  a certain  type  of  data. 

4.1.3  Best  Use 

These  characteristics  cover  several  aspects  of  technique  comparison.  Cer- 
tain techniques  may  be  considered  as  best  for  gathering  either  subjective 
or  objective  data.  In  most  cases,  these  amount  to  qualitative  or  quantita- 
tive data  respectively.  However,  some  subjective  data  is  quantitative  (num- 
erical ratings)  and  some  objective  data  is  purely  qualitative  (go/no  go  con- 
ditions) . 

Some  techniques  are  best  suited  for  pure  testing  as  opposed  to  pure  evalua- 
tion. For  example,  the  end  product  of  some  techniques  is  simply  readouts 
of  alphanumeric  data.  Other  techniques  are  needed  to  analyze,  evaluate, 
or  draw  conclusions  from  the  raw  data  originally  available. 

Many  techniques  are  best  used  for  only  a portion  of  the  total  system  being 
evaluated.  They  may  be  useful  for  examining  the  HFE  aspects  of  a component, 
but  much  less  useful  for  a subsystem  evaluation,  and  not  of  any  value  for 
a complete  system.  In  a similar  manner,  some  techniques  are  best  used  for 
performing  T&E  on  only  a single  task  performed  by  one  test  participant. 

Other  techniques  work  well  on  several  tasks  being  performed  nearly  simul- 
taneously by  several  test  participants. 

4.1.4  Relative  Performance 

These  characteristics  pertain  to  the  ease  of  using  one  technique  versus 
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another  in  terms  of  time  to  perform,  complexity,  personnel  required,  and 
cost.  The  relative  time  required  to  perform  a technique  for  a given  test 
may  be  described  as  short,  medium,  or  long.  The  time  factor  may  be  signifi- 
cant depending  on  test  schedules.  The  complexity  characteristic  is  impor- 
tant if  the  skills  of  the  HFE  test  observer  personnel  is  limited  by  exper- 
ience. The  number  of  personnel  is  also  significant  if  the  HFE  organization 
is  limited  and/or  other  important  tests  must  be  observed  simultaneously. 

The  cost  is,  of  course,  one  of  the  most  important  of  all  the  test  evalua- 
tion characteristics. 

‘1.1.5  Program  Interface  Requirements 

These  characteristics  include  the  input  and  outputs  to/from  the  techniques 
and  the  coordination  required  in  order  to  use  the  technique.  Prior  to 
choosing  a technique,  it  is  important  to  know  if  the  proper  ingredients  are 
available  to  initiate  the  technique.  It  is  also  important  to  know  if  the 
outputs  will  be  what  is  desired  or  in  the  required  format  (Reference  10). 
Special  test  coordination  as  to  other  organizations  or  support  that  is  neces- 
sary to  perform  the  test  must  also  be  considered  in  choosing  HFE  T&E  tech- 
niques. 

The  need  for  a particular  T&E  technique  to  be  performed  on  a test  "interfer- 
ence  basis"  is  extremely  important.  An  interference  basis  test  is  defined  as 
one  which  allows  very  little  other  testing  (HFE  or  any  other)  to  be  conducted 
simultaneously.  The  conduct  of  formal  T&E  is  a relatively  expensive  opera- 
tion. It  is  therefore  desi rabl e to  have  as  much  HFE  T&E  as  possible  be  con- 
ducted during  the  performance  of  other  testing  on  a noninterference  basis. 

In  many  instances,  more  realism  may  be  obtained  if  the  HFE  testing  is  per- 
formed along  with  the  conduct  of  other  major  system  tests. 

The  advantage  of  tests  conducted  for  HFE alone  are  in  the  ability  to  more 
easily  control  test  variables  and  to  study  a wider  range  of  test  conditions. 
However,  such  testing  for  HFE  alone  is  more  expensive.  One  alternative  to 
noninterference  testing  would  be  in  the  performance  of  static  or  "power-off" 


testing  which  could  be  performed  by  HFE  during  the  periods  of  time  that  the 
test  equipment  or  mockups  are  not  being  used  by  any  other  test  organizations. 

The  use  of  the  design  criteria  checklist  is  an  example  of  such  a test  tech- 
nique that  could  be  used  in  this  manner. 

4.2  General  Technique  Categories  \ 

The  various  HFE  T&E  techniques  are  categorized  for  the  purpose  of  Reference 
and  comparison  with  the  previously  indicated  technique  evaluation  character- 
istics. Examination  of  the  various  technique  types  leads  to  the  realization 
that  two  major  categories  are  automatic  and  manual.  The  automatic  techniques 
rely  primarily  on  test  instrumentation  or  computer  simulations.  The  manual 
techniques  rely  to  a large  extent  on  the  observer  as  the  data  collection  de- 
vice. Figure  4.2-1  shows  each  of  the  T&E  methodologies  in  relation  to  their 
major  categorizations. 

4.2.1  Manual  Techniques 

A further  distinction  in  the  manual  category  is  that  of  direct  versus  indirect 
observation.  System  measurement  is  considered  a third  category  of  manual  tech- 
nique; although  it  could  also  be  classified  as  a form  of  indirect  observation. 
Direct  manual  techniques  may  be  further  categorized  as  being  continuous  or  sam- 
pled. System  measurement  techniques  are  where  the  test  evaluator  gathers  HFE 
T&E  data  through  a review  of  test  logs,  maintenance  records,  or  debriefing 
records  to  obtain  the  necessary  HFE  T&E  data.  Indirect  techniques  include 
interviews  and  questionnaires.  The  technique  descriptions  in  Section  5.0 
provide  considerably  more  detail  in  the  way  of  individual  technique  comparisons. 

4.2.2  Automatic  Techniques 

Automatic  techniques  are  categorized  as  to  those  that  are  primarily  measure- 
ment devices  of  external  data  and  those  that  both  measure  and  create  new  data 
based  on  specified  inputs.  This  later  category  consists  of  the  computer 
program  techniques  or  computer  simulations.  These  may  be  further  divided 
into  on-line  interactive  techniques  and  off-line  (e.g.,  batch  mode)  techniques. 

The  measurement  devices  include  all  of  the  recording  techniques/tools  such  as 
video  tapes,  sound  tapes,  event  recorders,  and  photography.  Measurement 
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igure  4.2-1:  Categories  of  T&E  Techniques 


devices  also  include  physiological  instrumentation  of  parameters  such  as 
heart  rate  and  EEG,  Also  included  under  measurement  devices  are  the  phys- 
ical anthropological  techniques/tools  such  as  the  tape  measure  and  goni- 
ometer. 

4.3  Inherent  Technique/Characteristic  Relationships 

The  particular  relationships  between  techniques  and  evaluation  character- 
istics may  be  found  in  Section  5.0  and  in  Appendix  D.  There  are,  however, 
some  general  relationships  that  may  be  stated.  One  general  relationship 
is  somewhat  similar  to  the  relationship  noted  between  man  (manual)  and 
machines  (automatic)  in  the  notable  Fitts  list  (Reference  3). 

Manual  techniques  tend  to  be  more  subjective  and  qualitative  than  automatic 
techniques.  Their  cost  tends  to  be  lower  than  automatic  techniques.  Auto- 
matic techniques  are  better  used  for  objective  and  quantitative  data.  The 
validity  or  reliability  of  automatic  techniques  tends  to  be  higher  than  man- 
ual techniques.  This  is  because  manual  techniques  are  often  based  on  the 
opinion  of  the  test  participant  or  observer.  Automatic  techniques  are  based 
on  actual  measured  or  precisely  calculated  data.  Automatic  techniques  tend 
to  be  more  costly  and  more  complex  than  manual  techniques.  Automatic  tech- 
niques require  more  time  to  initiate  but  once  set  up,  they  are  more  effi- 
cient than  manual  techniques.  Simulations  are  best  used  early  in  a program. 

Another  general  relationship  between  techniques  is  the  inverse  relationship 
between  their  fidelity  (or  accuracy)  and  their  ease  of  use.  The  end  product 
of  any  T&E  effort  is  to  make  valid  judgments  in  regard  to  the  real  world  sit- 
uation. Some  techniques  are  inherently  more  precise  than  others  in  terms  of 
their  fidelity  to  the  proposed  actual  operating  system  situation.  Unfortu- 
nately, the  reason  for  their  fidelity  is  largely  due  to  the  effort  required 
by  the  use  of  the  technique.  This  inverse  relationship  between  technique 
accuracy  and  ease  of  use  is  illustrated  in  Figure  4.3-1.  The  real  world  sit- 
uation is  represented  at  the  left  of  the  figure.  The  most  accuracy  would  be 
obtained  by  always  measuring  the  real  world  situation.  However,  as  always, 
the  process  of  making  observations  or  measurements  tends  to  distort  the  test 
results  and  a degree  of  test  fidelity  is  lost. 


39 


THE 

REAL 

WORLD 


OBSERVATIONS  ANOl 
MEASUREMENTS  IN  W 
THE  REAL  WORLD  j ' 

h* FIELD  STUDIES- 


h 


SIMULATIONS  AND  MOCKUPS 


LABORATORY  EXPERIMENTS 
COMPUTER  . 


MODELS 


HIGH 
LOW  ■ 


FIDELITY  OF  TEST  CONDITIONS  ■ 
- EASE  OF  PERFORMING  TESTS  - 


LOW 

HIGH 


Figure  4.3-1 , Technique  Fidelity  Vs.  Ease  of  Use 


Field  studies  have  less  fidelity  than  real  world  observations  because 
such  studies  are  set  up  as  artificial  situations.  Simulations  with  the 
use  of  mockups  cover  a wide  range  on  the  fidelity  scale.  Laboratory  ex- 
periments also  have  a wide  range  of  fidelity  variability.  Some  labora- 
tory experiments  may  be  quite  realistic  while  others  may  be  highly  ab- 
stract. Computer  or  mathematical  models  are  the  easiest  to  develop, 
manipulate,  and  use,  but  they  are  the  least  accurate  in  their  represen- 
tation of  real  world  situations.  Considerable  advancement  is  now  being 
made  in  the  area  of  developing  computer  models  to  simulate  more  accurately 
real  world  situations. 

4.4  Drawings,  Mockups  and  Prototypes 

There  are  a number  of  different  tools  or  media  that  engineers,  in  general, 
and  HF  engineers,  in  particular,  work  with  throughout  the  life  of  a program. 
These  range  from  engineering  sketches  during  the  exploratory  or  concept 
formulation  phase  of  a program  to  the  prototype  hardware  which  would  result 
from  a full  scale  development  effort.  The  techniques  that  the  HFE  observer/ 
evaluator  uses  depend  to  some  extent  upon  the  media  or  engineering  tools 
available. 
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The  first  tools  with  which  the  HFE  has  to  work  during  the  "t&e",  or  explor- 
atory, phase  of  the  program  are  engineering  sketches.  These  sketches  may 
be  well  crafted  engineering  drawings  but  are  referred  to  as  sketches  only 
because  of  their  intended  lack  of  contractor  of  customer  sign-off  approval. 
Interface  control  drawings  are  another  type  of  drawing  that  should  require 
HFE  review.  As  the  name  implies,  these  drawings  are  used  to  describe  and  to 
eventually  control  proposed  interfaces  between  components,  subsystems,  or 
different  contractor's  equipment  items.  Most  of  these  types  of  drawings 
may  be  used  to  perform  a rough  HFE  evaluation,  particularly  by  use  of  the 
design  criteria  checklist.  Software  and  equipment  item  specifications  are 
prepared  to  do  document  required  system  component  performance  and  design 
criteria.  To  a limited  extent,  these  may  be  used  to  verify  compliance  with 
contractual  specifications.  HFE  T&E  computer  simulation  techniques  may  be 
used  to  test  system  design  concepts  as  indicated  in  the  drawings  and  equip- 
ment item  specifications. 

As  the  system  design  progresses,  full-scale  mockups  should  be  constructed 
to  evaluate  workspace  and  accessibility  provisions.  Wiring,  cabling,  pip- 
ing, and  ducting  may  be  designed  and  evaluated  much  more  easily  with  mock- 
ups.  It  is  difficult  to  visualize  three-dimensional  problems  from  scaled 
down,  two-dimensional  drawings.  The  mockups  should  be  made  initially  with 
the  easiest  to  use  and  cheapest  material  possible.  Various  thickness  plas- 
tic foam  core  filled  cardboard  sheets  may  be  used  quite  easily  with  a hot 
glue  gun  and  a sharp  matte  knife  to  build  consoles,  racks,  and  even  complete 
cockpits.  Console  panel  layout  drawings  may  be  simply  glued  to  the  foam 
core  cardboard  to  simulate  the  appropriately  located  displays  and  controls. 
Test  participants  or  evaluators  may  simulate  the  observation  of  displays 
or  actuation  of  controls  by  simply  touching  the  drawing  and  performing  the 
appropriate  hand  (foot)  motion.  As  the  system  design  progresses  and  mockup 
tolerances  become  more  critical,  plywood  material  should  be  used.  Plywood 
is  both  more  rigid  and  durable,  although  considerably  more  costly  in  terms 
of  construction  costs.  The  plywood  mockups  may  be  converted  from  a static 
representation  of  the  system  to  a dynamic  or  hot  mockup.  These  mockups  are 
also  referred  to  as  functional  mockups.  The  console  panel  drawings  which 
were  glued  to  the  plywood  may  be  replaced  by  the  actual  displays  and  con- 
trols. It  is  cheaper  to  develop  a hot  mockup,  which  includes  the  proposed 


41 


electrical  wiring,  than  it  is  to  build  a prototype  with  numerous  design 
errors.  A functional  mockup  makes  it  possible  to  study  the  performance 
of  personnel  in  simulated  operational  situations.  The  HF  engineer  can 
thereby  evaluate  operating  characteristics  of  equipment  in  terms  of  human 
performance.  More  realistic  lighting  and  sound  measurements  may  be  taken. 
Procedures  may  be  verified.  Test  participants  may  be  observed  and  inter- 
viewed with  a much  greater  degree  of  confidence  as  to  the  validity  of  their 
responses.  In  addition  to  all  of  the  above,  mockups  along  with  photographs 
and  movies  provide  a means  of  design  documentation  to  show  the  evolution 
of  the  system  configuration. 

The  last  major  engineering  tool  short  of  the  production  hardware  is  the 
prototype.  This  is  the  first  item  of  hardware  produced.  Whereas  it  is 
the  most  valid  representation  of  the  proposed  system  end  item,  it  does 
not  necessarily  comply  with  all  system  requirements.  Depending  on  the  ob- 
jective of  the  system  design,  the  prototype  hardware  may  be  the  first 
opportunity  to  evaluate  the  design  in  a mobile  state.  With  the  exception 
of  the  computer  simulation  techniques,  virtually  all  of  the  HFE  T&E  tech- 
niques may  be  used  for  prototype  hardware  man/machine  evaluation. 
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5.0  T&E  TECHNIQUES 


This  section  contains  details  as  to  the  description,  use  or  validity,  ref-  ] 

erences,  and  in  some  cases,  samples  of  33  significant  HFE  T&E  techniques,  1 

They  are  listed  according  to  the  categorization  presented  in  Section  4.2. 

5.1  Direct  Manual 

5.1.1  Direct  Continuous  Observation 

Description:  This  technique  is  simply  the  process  of  taking  a relatively 

continuous  record  of  the  task  or  work  activity  or  some  aspect 
of  the  test  performance.  The  operation  may  consist  of  an 
observer  keeping  a running  log  or  description  of  the  test 
activity  as  he  understands  it.  The  data  may  be  recorded  by 
hand  on  a clip  board,  or  some  of  the  more  sophisticated  tech- 
niques/tools (Section  5.4.5)  may  be  used  for  recording  events 
and  times.  Automatic  recording  techniques  such  as  photo- 
graphs, movies,  and  sound  and  video  tapes  (Section  5.4.3)  may 
also  be  used  along  with  direct  observation. 

The  detail  of  the  observed  data  is  in  accordance  with  the  basic 
considerations  indicated-  in  Section  2.4.  The  observer  should 
be  skilled  at  being  able  to  discriminate  what  significant 

events  occur  during  the  test.  These  events  should  be  summarized  r 

and  interpreted  for  later  action.  The  observer  must  be  familiar 

with  the  anticipated  man/machine  system  performance.  He  will 

observe  test  participants  while  they  are  using  either  mockups 

or  actual  hardware.  The  observer  should  be  particularly  . 

interested  in  obtaining  data  on  operator  task  times  and  errors. 

Data  as  to  the  observer's  estimates  of  participants'  training, 
skills  and  quantities  should  also  be  recorded.  Life  support, 
safety  and  hardware  design  criteria  compliance  may  also  be 
observed. 

The  use  of  the  direct  observation  technique  involves  the  use 
of  mockups  or  hardware.  Therefore,  the  most  appropriate  time 


to  use  this  technique  would  be  any  time  after  the  system 
concept  has  evolved  sufficiently  to  produce  three  dimensional 
mockups. 


Use/Validity:  Observation  is  one  of  the  most  common  methods  of  evaluating 

personnel  and  system  performance.  It  is  used  to  some  extent 
in  some  form  in  every  test  and  evaluation.  Despite  the  increas- 
ing use  of  automatic  recording  devices  the  requirement  for 
direct  observation  will  never  be  completely  eliminated. 

Observation  may  be  used  on  any  portion  of  a total  system,  a 
subsystem,  or  on  system  components.  It  is  useful  -for  T&E  of 
single  task  performance  or  the  simultaneous  operation  of 
several  tasks  by  several  test  participants.  It  is  simple  to 
perform  in  comparison  with  other  T&E  techniques. 

The  major  advantages  of  the  direct  continuous  observation 
technique  are  two-fold:  the  observer  is  forced,  by  way  of  his 

observation,  to  learn  about  the  system  operation;  the  observer 
gets  to  know  the  test  participants,  which  may  be  helpful  with 
possible  later  interviews. 

During  the  conduct  of  the  test  it  is  possible  for  the  observer 
to  do  more  than  simply  record  test  occurrences.  The  observer 
may  evaluate  test  data  for  possible  recommendations  or  test 
action  items.  If  direct  continuous  observation  is  not  used, 
there  is  a risk  of  missing  an  overall  impression  of  the  test 
as  well  as  random  test  events  or  details  that  would  otherwise 
be  overlooked. 

One  of  the  disadvantages  of  using  this  technique  is  the  require- 
ment for  specialized  observers  for  each  of  the  different  test 
aspects  or  categories.  It  is  seldom  possible  for  a single 
observer  to  learn  a sufficient  amount  about  all  system  aspects 
to  perform  an  adequate  job  of  observing  all  system  tests. 
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The  use  of  continuous  observation  implies  some  periods  of 
test  observation  that  are  not  productive  in  terms  of  gathering 
HFE  T&E  data. 

a)  Air  Force,  Test  Plan  for  Category  II  Testing  of  System  416M, 
Prepared  by  the  416M  Test  Force,  Hanscom  AFB,  November  1963. 

b)  Alter,  F.  H.,  Ground  Electronics  System  for  WS-133B  (Minuteman), 
PSTE  Plan,  MOP  PD0060,  Sylvania  Electronic  Systems,  Minuteman 
Program  Office,  Waltham,  Mass.,  March  1964. 

c)  Lathrop,  R.  C. , et  al , Evaluation  of  the  HF  Aspects  of  the 
GAM- 77  (Hound  Dog),  APGC-TN-60-1 9,  AD  236953,  HF  Office,  Air 
Proving  Ground  Center,  Eglin  AFB,  April  1960. 

d)  Martin  Company,  PSTE  Test  Cycle  Report  on  Missile  SM68-11, 
CR-63-43,  AD  405  382,  Denver,  Colorado,  February  1963. 

e)  Martin-Marietta,  Titan  II  Category  II  Observer/Evaluator 

Handbook:  PSTE  Operating  Procedures:  Maintenance,  Logistics, 

etc,  for  SM-688,  Titan  II,  Denver,  Colorado,  July  1963. 

f)  Meister,  D.,  and  G.  F.  Rabideau,  1965.  Human  Factors  Evalua- 
tion in  System  Development.  New  York:  Wiley. 

Additional  data  on  this  technique  as  compared  with  other  manual 
observation  techniques  may  be  found  in  Appendix  D. 

5.1.2  Direct  Sampled  Observation 

Description:  This  technique  is  identical  to  the  previously  listed  one  with  the 

exception  of  the  amount  of  time  spent  by  the  observer  observing 
the  test.  The  particular  times  chosen  to  perform  test  observa- 
tion should,  of  course,  be  those  which  coincide  with  the  perfor- 
mance of  critical  tasks.  The  determination  of  anticipated 
critical  tasks  should  be  made  on  the  basis  of  the  program's 
preceding  systems  analysis  effort.  Random  sampling  for  T&E 
data  may  be  performed  if  possible  critical  tasks  have  noi  been 
predicted  by  analysis. 
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Use/Val idity:  The  only  difference  in  the  use  or  validity  of  the  sampled 

observation  technique  as  compared  to  continuous  observation 
is  in  cost  savings  and  the  risk  of  missing  significant  T&E 
data.  It  stands  to  reason  that  if  the  tests  are  not  observed 
continuously  the  test  observers  may  be  used  to  perform 
other  HFE  T&E  tasks  on  other  tests  or  in  data  reduction 
and  evaluation  of  previously  conducted  tests.  The  number  of 
personnel  required  to  perform  HFE  T&E  may  be  cut  by  a factor 
of  one  half  or  more.  The  disadvantage  of  the  sampling  technique 
is  in  running  the  risk  of  missing  important  operator  perfor- 
mance data  or  other  important  HFE  related  data.  If  critical 
tasks  cannot  be  predetermined,  test  sampling  should  be  performed 

with  relative  frequency.  All  basic  categories  or  types  of 
0 . 

‘ opera tor/ equipment  tasks  should  be  observed  several  times  in 
order  to  prevent  skewed  data. 

Reference:  Meister,  0.,  and  Rabideau,  G.  F.,  Human  Factors  Evaluation  in 

System  Development.  New  York:  Wiley,  1965. 

Additional  data  on  this  technique  as  compared  with  direct  con- 
tinuous observation  and  other  manual  observation  techniques 
may  be  found  in  Appendix  D. 

5.1.3  Design  Criteria  Checklist 

Descri ption:  The  checklist  is  a series  of  equipment  and  facilities  design 

requirements  or  criteria  taken  from  human  engineering  standards, 
e.g.,  MIL-STD-1472,  handbooks  and  guides.  Often,  during  the 
early  stages  of  a program,  a checklist  is  developed  by  HF 
engineers  for  that  particular  program.  Design  criteria  which 
would  be  applicable  to  the  particular  program  are  extracted 
from  the  various  standards  and  handbooks  and  listed  in  a 
program  unique  checklist.  The  checklist  may  be  divided  up 
into  sections  or  categories  of  design  criteria  corresponding 
to  major  equipment  or  facilities  characteristics.  These 
categories  might  be  visual  displays,  audio  displays,  controls. 
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etc.  The  checklists  generally  have  a space  to  the  right  of 
each  listed  item  of  design  criteria.  This  space  is  divided 
into  three  columns:  compliance,  non-compliance,  and  not 

applicable.  The  HFE  evaluator  reads  the  item  of  criteria, 
observes  the  item  of  hardware  (or  mockup  or  drawing),  and 
checks  the  appropriate  space  for  applicability  and  compliance. 
Many  checklists  provide  additional  space  to  include  comments 
as  to  the  reason  for  non-compliance  or  other  remarks  appro- 
priate to  the  listed  design  criteria  item. 

H 

The  HFE  evaluator  should  initiate  the  use  of  the  checklist 
with  at  least  some  knowledge  of  the  purpose  or  function  of 
the  design  item  being  evaluated.  He  must  have  a good  working 
knowledge  of  the  checklist  criteria  which  he  will  be  using. 

He  should  determine  if  the  item  of  hardware  has  had  any 
previous  checklists  completed  on  it,  even  if  the  hardware 
was  only  in  drawing  form  at  the  time.  The  more  formal  test 
and  evaluation  procedure  will  occur  when  the  item  being 
evaluated  is  at  least  in  the  prototype  hardware  stage  of 
development.  Less  formal  checklist  test  and  evaluation  may 
take  place  with  hardware  drawings  or  possibly  mockups.  In 
any  case,  the  evaluation  should  take  place  on  a non-interference 
basis,  i.e.,  the  gathering  of  the  checklist  data  should  not 
interface  with  the  conduct  of  any  other  test  aspects.  The 
use  of  the  checklist  is  essentially  a static  operation,  as 
opposed  to  a dynamic  test  which  requires  observation  of  opera- 
tors performing  their  tasks  and  equipment  properly  responding 
to  their  manipulation. 

The  checklist  evaluation  will  result  in  a verification  of  the 
fact  that  the  design  item  meets  all  pertinent  HE  design 
criteria.  If  some  design  criterion  is  found  not  in  proper 
compliance,  then  this  information  will  be  provided  to  design 
engineering  personnel.  In  some  situations  there  may  be 
satisfactory  rationale  as  to  why  an  item  of  hardware  does 
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not  or  should  not  meet  the  HE  design  requirements.  In  this 
case  a request  for  deviation  to  HE  design  criteria  may  be 
submitted  to  the  Navy  system  program  office  for  their 
approval . 

Use/Validity:  This  technique  is  used  more  often  than  any  other  to  evaluate 

design  hardware.  It  is  an  excellent  way  to  gather  quickly 
qualitative  data  on  system  hardware  components.  However,  in 
order  to  be  of  real  value,  there  must  be  considerable  detail 
contained  within  the  checklist.  Depending  upon  how  the 
checklist  is  structured,  the  amount  of  detail  required  for 
review  can  extend  the  time  required  to  perform  the  checklist. 

Use  of  the  checklist  requires  more  knowledge  of  basic  HE  design 
criteria  than  system  performance  requirements. 

The  disadvantages  associated  with  the  use  of  the  checklists 
are  that  they  produce  binary  data;  the  design  criteria  being 
verified  is  either  in  compliance  or  not.  However,  many 
criteria  items  have  the  potential  for  an  exact  quantitative 
evaluation;  thus  considerable  data  will  be  unrecorded.  The 
checklist  is  used  for  evaluation  of  hardware  only.  In  its 
present,  generally  agreed-to  formats,  the  checklist  will  not 
evaluate  personnel  skills,  quantities,  training,  technical 
publications,  etc. 

The  use  of  this  particular  technique  is  strongly  advised  for  all 
DT&E  programs.  If  not  used,  there  is  significant  risk  that 
lack  of  critical  design  compliance  requirements  will  be  over- 
looked. It  is  also  recommended  that  the  HFTEMAN  (Section  5.1.7) 
be  reviewed  prior  to  developing  any  new  program  unique  check- 
lists. 

References:  a)  Air  Force,  Personnel  Subsystems  Speci fication.  SAMSO 

Exhibit  68-19,  September  1968. 
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b)  Air  Force,  Test  Plan  for  Category  II  Testing  of  System  416M. 
Prepared  by  41 6M  Test  Force,  Hanscom  AFB,  November  1963. 

c)  Army,  Human  Factors.  Army  Test  and  Evaluation  Command,  Aber- 
deen Proving  Grounds,  MTP-6-3-525,  AD  728  824,  July  1971. 

d)  Army,  Human  Factors  Engineering.  Army  Test  and  Evaluation 
Command,  Aberdeen  Proving  Grounds,  MTP-2-3-516,  AD  871  909, 
May  1970. 

e)  Army,  Human  Factors  Engineering,  Army  Test  and  Evaluation 
Command,  Aberdeen  Proving  Grounds,  MTP-5-2-545,  AD  876  199, 
July  1970. 

f)  Hattendorf,  G.  A.,  et.  al.,  C-141A  Category  II  Systems  Eval- 
uation, Technical  Report  65-38,  Air  Force  Flight  Test  Cen- 
ter, Edwards  AFB,  July  1966. 

g)  Meister,  D.,  Human  Factors:  Theory  and  Practice.  New  York: 
Wiley,  1971. 

h)  Myers,  L.  B.,  et.  al..  Guidebook  for  the  Collection  of  Human 
Factors  Data,  Report  PTB  66-3,  AD  631  023,  HRB-Singer,  Inc., 
Pennsylvania  State  College,  January  1966. 

i)  Root,  R.,  Performance  Measures  Report,  Air  Traffic  Control 
Communications  System  AN/TSQ-47,  CR-62-548-9,  Defense 
Electronics  Products,  RCA,  Burlington,  Mass.,  November  1962. 

j)  "Human  Engineering  Design  Checklist",  WDL-TR-1968A, 

AD  829  426,  Philco-Ford  Corp. , Western  Development  Lab., 

May  1964. 

Additional  data  on  this  technique  as  compared  with  other  manual 

observation  techniques  may  be  found  in  Appendix  D. 

Sample:  See  following  page  (Figure  5.1-1). 

5.1.4  Specification  Compliance  Summary  Sheet 

Description:  This  is  a form  that  is  used  to  verify  that  system  performance 

is  in  accordance  with  specified  HFE  requirements.  Briefly,  the 
total  process  of  verifying  HFE  specification  compliance  is: 
first  to  decide  the  best  method  to  verify  the  specifica- 
tion requirement  (i.e.,  analysis,  demonstration,  or 
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quantitative  data),  second  to  perform  the  analysis/test  and 
third  to  document  the  results.  In  any  case,  reports  are 
written  as  to  the  analysis  or  test  results.  The  Specifica- 
tion Compliance  Summary  Sheet  is  a way  of  summarizing  this 
compliance  or  lack  of  compliance. 

The  evaluator  needs  first  to  have  a thorough  knowledge  of 
all  HFE  aspects  of  the  contract  statement  of  work  and  the 
accompanying  system  specifications.  In  particular,  he  should 
understand  the  specification  Section  4.0  requirements  (quality 
assurance/testing). 

After  the  test,  demonstration,  or  analysis  has  been  per- 
formed and  reported,  the  summary  sheet  form  is  completed. 

The  form  contains  a space  to  indicate  the  specification 
number  and  complete  section  being  verified.  Space  is  provided 
for  a summary  of  the  test/analysis  results.  Signature  blocks 
are  provided  for  persons  preparing  the  summary  sheets  and 
approving  the  verification  of  specification  performance. 

Use/Validity:  This  technique  is  used  by  only  a few  HFE  T&E  organizations. 

However,  this  lack  of  use  is  not  an  indication  of  the  need 
for  this  type  of  evaluation.  The  contract  and  related  system 
specifications  are  by  far  the  most  important  program  require- 
ments (Section  2.1).  This  technique  is  unique  in  that  it 
zeros  in  on  these  important  requirements,  rather  than  concerning 
itself  with  T&E  of  indirect  system  requirements. 

The  Specification  Compliance  Summary  Sheet  is  an  excellent 
way  to  verify  the  Section  4.0  specification  requirements.  The 
only  disadvantages  associated  with  the  use  of  this  form  are 
in  the  large  amount  of  time  required  to  fill  it  out.  The 
effort  preceding  the  use  of  this  form  may  be  considerable 
but  that  effort  is  a part  of  the  already  existing  HFE  T&E 
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program.  If  this  technique  is  not  used,  there  is  a risk  that 
some  important  aspect  of  HFE  design  criteria  may  be  overlooked 
both  by  designers  and  by  test  observers. 


Reference:  Additional  data  on  this  technique  as  compared  with  other 

manual  observation  techniques  may  be  found  in  Appendix  D. 

5.1.5  Technical  Order  Functional  Evaluation 

Descri ption:  As  its  title  would  indicate,  this  technique  is  designed  to 

evaluate  technical  orders  or  publications  pertaining  to  the 
test.  The  technique  is  based  on  the  use  of  a form  to  be 
completed  by  the  test  observers  while  they  are  performing 
their  other  direct  observations  of  the  test.  The  technical 
publications  must  be  evaluated  as  to  their  usefulness  and 
adequacy  in  three  areas: 

a)  Job  Instructions 

b)  Training 

c)  Job  Performance  Aids 

Job  Instructions  tell  how  to  accomplish  a task  by  providing 
the  step-by-step  procedures  along  with  the  necessary  illus- 
trative drawings. 

Most  technical  publications  which  require  validation  or 
verification  provide  support  for  training. 

There  are  three  major  types  of  job  performance  aids  identified 
as  follows: 

a)  Job  Guides  (including  inspection  guideline  manuals). 
These  guides  contain  instructions  for  fixed-procedure 
tasks  such  as  checkout,  adjustment,  removal,  and 
replacement. 

b)  Fully  Procedural i zed  Trouble  Shooting  Aids  spell 

out  the  steps  to  follow  in  isolating  malfunctions  to  a 


52 


f 


I 


replaceable  or  repairable  unit.  The  steps  start 
with  observable  symptoms  of  malfunction, 
c)  Troubleshooting  Decision  Aids  provide  diagrammatic 
and  supporting  textual  information  which  will  help  the 
technician  decide  what  steps  to  take  in  isolating 
malfunctions  to  a replaceable  or  repairable  unit. 

The  following  sample  evaluation  form  (Figure  b.1-2)  is  struc- 
tured so  that  the  first  three  questions  require  two  judgements: 
one  dealing  with  the  category  of  the  section  being  evaluated 
and  the  other  as  to  the  adequacy.  The  two  questions  are  to 
be  answered  by  the  test  evaluator/observer,  as  well  as  the 
test  participants.  The  remaining  questions  (4  through  7)  deal 
with  the  qualitative  characteristics  of  the  T.O. 

Most  sections  of  the  form  are  self-explanatory,  however,  the 
following  sections  should  be  completed  as  indicated: 

Evaluator:  Identify  individual (s)  interviewed  or  those 
contributing  to  the  evaluation. 

Paragraphs  Evaluated:  List  only  those  paragraphs  for 

which  the  evaluation  applies.  In  some  cases, 
this  can  be  done  in  large  blocks.  There  will 
be  some  events  where  several  separate  forms 
will  have  to  be  completed. 

T.O.  Verification  Personnel  Requirements:  When  verifica- 

tion is  performed,  the  names  and  rate  (rank) 
as  well  as  skill  code  of  the  participants  is 
required. 

Prior  to  conducting  this  type  of  evaluation,  the 
observer  or  evaluator  must  have  a knowledge  of 
the  technical  manual  he  is  to  evaluate.  He  must 
also  be  familiar  with  estimated  system  and  oper- 
ator/maintainer  performance.  The  total  technical 
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TECHNICAL  ORDER  FUNCTIONAL  EVALUAT)OM 

ASSESS  THE  USABILITY  OF  THE  IDENTIFIED  PARAGRAPHS  FOR  INSTRUCTIONS.  TRAINING,  AND/OR 
JOB  performance  aids.  (SEE  INSTRUCTIONS  ON  REVERSE  SIDE) 


T.O.  NUMBER:_ 
EVALUATOR: 


TITLE. 


DATE: 


PARAGRAPHS  OR  SECTIONS  EVALUATED:  (GIVE  NUMBER  AND  SUBJECT) 


NOTE:  PARAGRAPHS  OR  SECTIONS  COVERED  SHOULD  BE  CONSECUTIVE  AND  GROUPED  SO  THAT 

ANSWERS  GIVEN  APPLY  TO  ALL  PARAGRAPHS  LISTED. 

T.O.  VERIFICATION  PERSONNEL  REQUIREMENTS 


CONDITIONS  AT  VERIFICATION:  (INCLUDE  EQUIPMENT  involved.  WHERE  PERFORMED,  ETC.) 


EVALUATION;  (USE  additional  SHEETS  FOR  COMMENTS) 

1.  DO  THE  PARAGRAPHS  CONSITUTE  JOB  INSTRUCTIONS?  YES NO 

IF  YES,  ARE  THEY  ADEQUATE? 

2.  SHOULD  The  PARAGRAPHS  BE  USED  FOR  TRAINING?  YES N(D 

IF  YES,  ARE  THEY  ADEQUATE? 

3.  DO  THEY  CONSITUTE  JOB  PERFORMANCE  AIDS?  YES NO 

IF  YES,  ARE  THEY  ADEQUATE? 

4.  ARE  THE  STEPS  IN  LOGICAL  SEQUENCE  AND  DO  THE  ELIMINATE  BACK  TRACKING 
WHERE  POSSIBLE? 

5.  DID  the  individuals  DEMONSTRATING  THE  OPERATION  EXPERIENCE 
ANY  DIFFICULTY  AS  EVIDENCED  BY  ERRORS,  TOO  MUCH  TIME,  OR  NEED 
FOR  ASSISTANCE? 

6.  ARE  the  FUNCTIONS  DESCRIBED  SUF F ICIENTLY  NEW  OH  COMPLEX  AS  TO 
REQUIRE  TRAINING? 

7.  IS  IT  NECESSARY  TO  PROVIDE  ADDITIONAL  BACKGROUND  OR  SUPPLEMENTARY 
INFORMATION  IN  ORDER  FOR  THE  USER  TO  UNDERSTAND  WHAT?,  HOW?. 

WHEN?,  WHERE?  ETC.? 


Figure  5. 1-2:  Sample  Technical  Order  Functional  Evaluation  Form 
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Use/Validity 


References: 
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order  functional  evaluation  process  will  result 
in  either  verification  of  the  technical  data  or 
revisions  or  recommendations  for  new  technical 
data.  These  revisions  will  be  coordinated  with 
the  publications  writers. 

Depending  on  the  scope  or  charter  of  the  HFE  T&E  effort  tech- 
nical order  evaluation  may  or  may  not  be  performed.  If  it  is 
performed  (by  HFE  personnel),  it  may  be  accomplished  at  any 
time  with  the  evaluation  of  any  evolving  systems  (as  opposed 
to  future  or  existing  systems).  The  effort  required  to  per- 
form this  evaluation  is  relatively  low  and  it  is  therefore 
recommended  as  a task  to  be  accomplished  by  HFE  or  other 
personnel.  Failure  to  perform  this  evaluation  can  result  in 
several  maintenance  and  operational  mistakes  that  would  other- 
wise have  been  avoided.  The  cost  to  perform  the  evaluation 
must  be  considered  to  be  relatively  low,  particularly  compared 
to  the  potential  cost  of  the  mistakes. 

a)  Alter,  F.  H.,  Ground  Electronics  System  for  WS-133B 
(Minuteman) , PSTE  Plan,  MOP  PD  0060,  Sylvania  Electronic 
Systems,  Minuteman  Program  Office,  Waltham,  Mass., 

March  1964. 

b)  Lathrop,  R.  C.,  et  al , Evaluation  of  the  HF  Aspects  of 
the  GAM- 77  (Hound  Dog),  APGC-TN-60-1 9,  AD  236  953,  Human 
Factors  Office,  Air  Proving  Ground  Center,  Eglin  AFB, 

April  1960. 

c)  Martin  Co.,  PSTE  Test  Cycle  Report  on  Missile  SM  68-11, 
CR-63-43,  AD  405  382,  Denver,  Colorado,  February  1963 

d)  Mart in -Marietta,  Titan  II  Category  II  Observer/Evalua- 
tor Handbook:  PSTE  Operating  Procedures:  Maintenance, 

Logistics,  etc.,  for  SM-688,  Titan  II,  Denver,  Colorado, 
July  1963. 

e)  Potempa,  K.  W.,  "Automated  Readability  Index  to  Determine 
Reading  Difficulty,"  A Catalog  of  Human  Factors  Techniques 
for  Testing  New  Systems,  AFHRL-TR-68-15,  February  1969. 
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Sample: 

5.1.6  Human 
Description: 


Use/Validity 


Additional  data  on  this  technique  as  compared  with  other 
manual  observation  techniques  may  be  found  in  Appendix  D. 

See  evaluation  form  on  following  page. 

Performance  Reliability  Testing 

The  method  or  approach  employed  in  the  use  of  this  technique 
is  simply  direct  observation;  the  purpose  is  to  record  test 
participant  errors.  This  testing  may  occur  at  any  time 
during  direct  observation  testing.  As  in  the  case  of  the 
direct  observation  technique  (Section  5.1.1)  the  observer 
must  first  become  familiar  with  the  anticipated  man/machine 
performance. 

In  all  probability  the  test  observer  will  wish  to  develop 
categories  of  errors  to  assist  with  his  evaluation  of  the 
test  error  data.  Special  data  sheets  may  be  developed  for 
both  recording  and  categorizing  error  data  in  one  operation. 
The  most  valid  test  data  will  be  obtained  with  large 
Subject  or  participant  populations  and  when  several  test 
replications  are  performed.  As  in  any  test,  the  subjects 
should  be  trained  and  motivated  as  to  their  expected 
participation  in  the  test.  The  observer  may  wish  to  gather 
both  error  data  and  time  data  if  test  conditions  exist  that 
will  allow  this.  Several  observers  may  be  necessary  if 
several  test  subjects  are  used  and  it  is  necessary  that  all 
errors  are  noted. 

Most  human  performance  testing  involves  the  gathering  of  time 
data  from  test  participants.  Testing  for  operator/maintainer 
errors  tends  not  to  be  performed  because  of  the  problem  of 
not  knowing  what  standard  or  specified  error  criteria  to  test 
to.  That  is  to  say,  if  testing  determines  a particular  error 
rate,  there  is  no  way  of  knowing  if  that  error  rate  is  either 
bad  or  good.  Few  valid  human  performance  reliability  (HPR) 
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estimates  exist  today.  This  is  not  to  say  that  human  perform-  j 

ance  reliability  should  not  be  done.  On  the  contrary,  an 
accurate  data  base  of  field  generated  HPR  estimates  is  sorely 
needed. 

The  major  problem  in  gathering  or  generating  this  sort  of 
human  performance  data  is  to  derive  inmediate  use  or  benefit 
from  it.  If  only  a small  amount  of  HPR  data  is  obtained, 
it  cannot  be  usefully  compared  to  any  existing  data  banks. 

The  recommended  use  of  most  data  bank  data  is  misleading  and 
the  data  itself  is  erroneous.  If  a large  enough  amount  of  data 
is  gathered  to  be  useful,  the  cost  of  the  effort  must  be 
extremely  high.  A large  amount  of  HPR  data  is  necessary  to 
determine  accurately  the  effects  of  various  test  variables. 

An  HPR  of  .999  implies  at  least  1000  test  replications.  In 
many  cases,  the  effects  of  test  variables  will  not  be  noticeable 
until  four  significant  figures  are  obtained,  i.e.,  10,000  test 
replications/subjects . 

In  spite  of  the  high  costs  implicit  in  obtaining  HPR  test 
data,  it  is  to  be  encouraged.  More  emphasis  is  needed  on 
operator/maintainer  reliability,  if  extensive  HPR  testing  is 
not  possible,  error  data  should  still  be  recorded  with  the 
hope  that  experienced  evaluator  judgment  can  be  used  along 
with  the  limited  available  data  to  determine  significant 
error  effects.  This  effort  is  appropriate  to  MIL-H-46855, 
paragraph  3. 2. 4. 3,  "Failure  Analysis",  compliance. 

Reference:  McCalpin,  J.  P. , and  Miles,  J.  L.,  Determining  Human  Perform- 

ance Reliability  with  Infantry  Weapons : Part  One,  Technical 
Memorandum  22-74,  U.S.  Army  Human  Engineering  Laboratory, 

Aberdeen  Proving  Ground,  October  1974. 

Additional  data  on  this  technique  compared  to  other  manual 
observation  techniques  may  be  found  in  Appendix  D.  See 
also  Section  5.2.2,  Human  Initiated  Failures. 
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5.1.7  Human  Factors  Test  and  Evaluation  Manual  (HFTEMAN) 

Description;  HFTEMAN  must  be  considered  as  considerably  more  than  an 

HFE  T&E  technique.  It  is  a human  factors  test  and  evalua- 
tion manual  that  is  designed  to  assist  the  HF  engineer  in 
the  areas  of  test  plan  preparation,  test  conduct,  test  data 
evaluation  and  analysis,  and  test  report  preparation. 

The  HFTEMAN  consists  of  two  documents:  the  first  contains 

detailed  HFE  test  data  and  the  second  is  a guide  book 
supplement  that  contains  specific  HFE  design  criteria. 

The  procedure  of  using  HFTEMAN  may  be  considered  as  a five 
step  process.  This  procedure  is  well  detailed  on  the  first 
few  pages  of  the  manual.  The  first  step  requires  that  test 
items  be  classified  as  to  vehicles,  weapons,  electronics, 
etc.  The  second  step  is  to  identify  both  the  user  functions 
and  tasks  related  to  this  type  of  equipment;  in  other  words, 
a selection  is  made  of  what  to  evaluate  and  the  criteria  to 
be  used  in  the  evaluation  tests.  The  third  step  decides 
what  human  factor  considerations  and  what  item  components  are 
relevant.  The  test  observer  should  review  the  task  list 
and  test  item  design  description  to  identify  which  of  the 
test  item  components  presented  in  the  matrix  apply  to  the 
item  under  test,  and  which  human  factors  considerations 
are  important.  In  the  fourth  step  the  test  evaluator  goes 
from  the  cells  of  HF  considerations/task  item  components  to 
cells  containing  the  exact  test  criteria  as  indicated  on  a 
separate  (opposite)  page.  The  last  step  is  to  prepare  the 
HFE  test  plan  which  includes  an  "objective"  (taken  from 
HFTEMAN),  "criteria"  (taken  from  HFTEMAN) , and  "methodology" 
(taken  from  the  HFTEMAN  Supplement).  The  "data  required"  also 
is  provided  in  both  the  HFTEMAN  and  HFTEMAN  Supplement. 

It  is  recommended  that  the  test  observer  be  thoroughly  familiar 
with  the  HFTEMAN  contents  before  he  starts  this  procedure.  The 
end  products  of  this  effort  should  be  both  an  itemized  listing 
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of  all  HFE  system  deficient  items  and  a general  feeling  of 
pilot  or  other  operator  acceptance  of  the  hardware  item. 

Use/Val idi ty:  HFTEMAN  may  be  used  on  any  program  at  any  time  during  the 

program  evolution.  HFTEMAN  is  of  more  than  normal  value  in 
that  it  provides  both  the  basis  on  which  to  build  an  HE 
checklist  (Reference  Section  5.1.3)  and  all  of  the  rest  of 
the  necessary  HFE  T&E  planning  and  conduct. 

HFTEMAN  has  broad  applicability.  No  special  test  equipment 
is  required  to  use  with  this  technique  and  it  will  be  of  use 
with  any  military  system.  If  HFTEMAN  is  not  used,  the 
appropriate  HFE  test  planning  must  be  based  on  other  less 
coordinated  resources. 


HFTEMAN  has  derived  from  the  U.S.  Army  TECOM  Human  Factors 
Engineering  Data  Guide  for  Evaluation  (HEDGE).  The  Army 
guide  has  been  used  successfully  since  its  publication  in 
1974. 


Reference: 


a)  HEDGE:  Human  Factors  Engineering  Data  Guide  for 

Evaluation.  U.S.  Army  Test  and  Evaluation  Command, 
Aberdeen  Proving  Ground,  March  1974. 

b)  Kagerer,  R.  L.,  and  Weiss,  E.  C.,  Development  of  a 
Checklist  and  Guidebook  for  HF  Evaluation  of  General 
Equi pment.  Final  Report  30  June  66-1,  AD  827  808L, 

Matrix  Corp.,  January  1968. 

c)  Navy:  Human  Factors  Test  and  Evaluation  Manual  (HFTEMAN), 

Vol  I Data  Guide,  Vol  II  Support  Data,  Vol  III 

Methods  and  Procedures,  TP-76-IIA,B,C,  Pacific 
Missile  Test  Center,  Point  Unger,  CA,  April  1976. 

d)  Navy:  Human  Factors  Test  and  Evaluation  Manual  (HFTEMAN), 

Video  Tape,  Pacific  Missile  Test  Center,  Point 

Under,  CA,  1976. 


Additional  data  on  this  technique  as  compared  with  other 
manual  observation  techniques  is  contained  in  Appendix  D. 


5.1.8  G-2/G-5  Anthropometer 

Description:  These  are  two  anthropometric  devices  which  may  be  used  as 

models  of  various  size  or  percentile  crewmen  in  order  to 
check  the  adequacy  of  workstation  geometry.  They  may  be 
regarded  as  adjustable  representations  of  the  more  criti- 
cal crew  dimensions.  Each  device  is  more  analogous  to  a 
stick  figure  than  to  an  anthropometric  dummy.  They  may 
be  adjusted  to  the  95th,  98th  or  any  desired  percentile 
dimensions  for  several  of  the  more  important  crew  size 
parameters . 

Use/Validity:  The  G-2  anthropometer  may  be  used  at  any  crew  station. 

The  G-5  is  designed  primarily  for  use  as  an  escape  envel- 
ope measuring  device.  The  total  procedure  of  verifying 
critical  anthropometric  parameters  can  be  quite  tedious 
when  using  just  engineering  drawings  and  HE  design  cri- 
teria standards.  This  analytical  process  is  complex  and 
prone  to  errors.  The  use  of  these  devices  greatly  sim- 
plifies the  process  of  calculating  the  crewmen's  required 
reach  or  escape  envelopes.  The  use  of  actual  crewmen  to 
test  these  critical  envelopes  is  somewhat  impractical  be- 
cause of  the  difficulty  in  obtaining  personnel  with  the 
required  dimensions  for  each  of  the  pertinent  anthropo- 
metric parameters. 

Somewhat  similar  devices,  such  as  "compound  goniometers", 
have  also  been  developed  at  NADC's  Crew  Systems  organi- 
zation to  measure  (after  the  fact)  cockpit  geometry. 

References : a)  Gregoire,  H.  G.,  and  Barnes,  J.  R. , Cockpit  Anthropo- 

metric  Survey  of  Model  A-4C,  A-6A,  A-7E,  AV-8A,  F-4J, 
F-8D  and  OV-lOA  Airplanes,  NATC-ST-;20R-71 , AD  890  283L , 
Naval  Air  Test  Center,  July  1971. 
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5,1.9  Terrain 
Description: 


b)  Navy.  Investigation  of  A-4  Aircraft  Escape  System 
Clearance  Envelope,  NATC,  ST-53R-72,  AD  893  283L, 

Naval  Air  Test  Center,  Febraary-March  1972. 

Additional  data  on  this  technique  as  compared  with  other 
manual  observation  techniques  is  contained  in  Appendix  D. 

Visibility  Definition 

With  the  participant  (pilot)  seated  in  the  aircraft  (or  mc'ck- 
up)  cockpit  and  the  aircraft  in  a horizontal  flight  attitude 
direct  measurements  of  the  canopy  sill  vertical  and  horizori- 
tal  visual  angles  are  made  with  a surveyor's  transit.  The 
detailed  procedure  is  as  follows: 

1.  The  transit  is  leveled  in  front  of  the  aircraft  and  the 
over-the-nose  visual  angle  is  determined.  This  visual 
angle  is  the  angle  existing  between  the  horizontal  visir- 
line  and  the  down  vision  line  (see  Figure  5.1 -3a). 

2.  The  eye  line  is  established  from  the  down- vi si  on- line 
instrument  station  and  then  projected  and  drawn  on  the 
floor  under  the  aircraft  to  establish  an  eye  line.  The 
eye  line  will  fall  approximately  on  the  longitudinal  mid- 
line of  the  aircraft. 

3.  The  transit  is  then  moved  to  the  side  of  the  aircraft  and 
sighted  on  the  pilot's  eye.  The  transit  is  depressed  until 
it  intersects  the  eye  line  on  the  floor.  This  point  of 
intersection  is  designated  the  visual  reference  point  (VRP). 
See  Figure  5.1 -3b.  The  VRP  is  located  on  the  ground 
directly  beneath  the  eye  of  the  pilot. 

4.  The  pilot  then  uses  an  engineer's  hand  level  to  plot  the 
visual  points  over  the  sill,  from  0°  to  180°  positions. 

These  points  are  worked  on  the  floor  (ground)  by  a second 
person. 
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5.  The  transit  is  then  set  up  over  the  VRP,  and  the  hori- 
zontal visual  angle  is  then  recorded  for  each  viewable 
point  (see  Figure  5.1-3c).  The  distance  from  the  visual 
reference  to  each  viewable  point  is  also  measured,  ‘'his 
total  procedure  establishes  the  following  four  parameter . 
(see  Figure  5.1 -3d); 

a)  horizontal  angle 

b)  horizontal  distance  from  pilot's  eye  to  a marked  po'it 

c)  vertical  distance  from  pilot's  e>e  to  a marked  roinf 

d)  slant  range  from  oilot's  eye  to  ma'^ked  boint 

Use/Val idity:  Using  the  data  obtained  from  the  preceding  measurement  pro- 

cedures, the  following  parameters  can  be  calculated; 

a)  vertical  angles  for  the  visual  field  outside  the  air- 
craft (see  Figure  5.i-3d). 

b)  the  horizontal  and  vertical  plot  of  the  blind  zone 
and  viewable  area. 

c)  It  is  also  possible  to  develop  tables  for  various 
altitudes  which  can  be  used  to  compute  total  actual 
ground  area  visible  to  the  pilot  while  in  horizontal 
fl ight. 

This  technique  has  been  used  on  the  RF-101  and  RF-4C  aircraft, 
but  it  is  in  no  way  limited  to  existing  aircraft.  A mockup  of 
the  cockpit  configuration  is  sufficient  to  calculate  the 
necessary  visual  geometry  parameters.  Early  knowledge  of 
the  pilot's  (or  crewman's)  visual  envelope  is  essential  to 
determination  of  the  total  aircraft  system  performance. 

Without  the  use  of  this  technique,  it  would  be  unnecessarily 
difficult  to  compare  on  a quantified  basis  the  various  air- 
craft's cockpit  visual  field  capabilities. 
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HORIZONTAL  VISION  LINE 


B.  DETERMINING  THE  VISUAL  REFERENCE  POINT 

Figure  5. 1-3:  Terrain  Visibility  Definition 
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D.  DETERMINING  VERTICAL  ANGLES  FOR  THE  VISUAL 
FIELD  OUTSIDE  THE  AIRPLANE 


Figure  5. 1-3:  Terrain  Visibility  Definition  (Continued) 


Reference; 


Collins,  L.  R.  and  Benny,  J.  C. , Cockpit  Sill  Visual  Angle 
and  Terrain  Visibility  Definition  for  RF-101  and  RF-4C 
Aircraft,  McDonnell  Company  Report  F-179,  March  1967. 

Additional  data  on  this  technique  as  compared  to  other  manual 
observation  techniques  is  contained  in  Appendix  D. 

5.1.10  Environment  and  Performance  Measuring  Equipment 

Description  There  are  several  different  items  of  test  or  measuring  equip- 
ment  that  are  extremely  useful  to  the  HFE  test  observer.  A 
few  of  these  T&E  tools  are  presented  in  separate  sections, 
but  most  are  included  here.  The  following  subparagraphs  in- 
dicate the  item  of  HFE  test  equipment  along  with  a brief 
description  of  its  use; 

a)  Photometer.  Measures  ambient  illumination  over  a range 
of  levels  from  approximately  .005  to  25,000  foot-candles. 
This  is  an  extremely  useful  tool.  It  is  particularly 
valuable  for  verifying  spec? fication  compliance  with 
light  level  requirements.  Sophisticated  mockups  or  proto- 
type equipment/facilities  are  required  for  the  proper  use. 
Most  photometers  are  relatively  easy  to  use. 

b)  Spot  Brightness  Meter.  Measures  small  area  brightnesses 
in  foot-Lamberts  within  angles  of  approximately  one  degree 
or  less.  This  tool  is  most  useful  for  measuring  prototype 
hardware  disolay  brightness  such  as  from  LED's  or  CRT's. 
Specification  compliance  may  be  verified  with  the  spot 
brightness  meter. 

c)  Sound  Level  Meter  and  Analyzer.  Measures  sound  in  the 
approximate  range  from  10  to  150  dB  for  standard  weighted 
noise  curves.  The  analyzer  provides  octive  band  analysis 
for  the  more  critical  speech  range  center  frequencies. 
Specification  compliance  in  terms  of  noise  curves  and 
speech  interference  levels  may  be  verified  with  this 
equipment.  Hazards  to  test  personnel  may  be  checked  prior 
to  overexposure  conditions.  Most  sound  level  meters  are 
relatively  easy  to  use. 
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d)  Vibration  Meter  and  Analyzer.  Measures  amplitude  and 
frequency  components  of  complex  vibrations.  The  analyzer 
may  be  used  to  determine  amplitudes  at  selectable  fre- 
quency bands  in  a range  from  2.5  Hz  to  25  KHz.  Potential 
vibration  hazards  to  test  participants  may  be  checked 
before  actual  test  exposure.  Specification  compliance 
may  also  be  verified. 

e)  Thermometer.  Measures  air,  surface,  or  liquid  temperatures. 
May  provide  a digital  readout  in  either  Celsius  (centigrade) 
or  Fahrenheit.  Should  include  capability  for  attachment 

to  several  temperature  sensor  probes. 

f)  Anemometer.  Measures  local  air  flow  in  the  range  of  0 to 
1000  ft/minute.  This  device  is  most  useful  for  determining 
crew  comfort  conditions. 

g)  Hygrometer  or  Psychrometer.  Measures  relative  humidity 
using  the  wet  and  dry  bulb  thermometer  method.  This  device 
is  also  very  useful  for  determining  conditions  for  crew 
comfort. 

h)  Gas  Tester.  Permits  convenient  short-term  sampling  and 
evaluation  of  many  toxic  gases,  vapors  and  fumes. 

i)  Force,  Torque  and  Dimension  Kit.  Various  instruments 
for  measurement  of  a wide  variety  of  operator  or  equipment 
forces,  torques  and  distances.  The  force  measurement  limits 
should  be  from  1/4  oz.  to  250  lbs.  Torque  measurement 
should  range  from  1/2  in-lb  to  160  ft-lbs.  A tape  measure 
should  be  caoable  of  measuring  distances  up  to  50  feet. 
Scales  should  also  be  for  measuring  centimeters,  milli- 
meters, inches  and  fractions  of  inches.  A protractor 

is  useful  for  angular  measurement. 

j)  Anthropometry  Instrument  Kit.  Allows  measurement  of  signi- 
ficant body  dimensions  using  the  arthropometer , spreading 
calioers,  sliding  caliper,  gonometer  and  tape  measure. 

The  measurement  of  test  participants  is  critical  to  the 
evaluation  of  workspace  layouts,  particularly  when  egress 
and  ingress  are  important  considerations.  Care  should  be 
taken  to  insure  the  proper  measurement  procedures  are 

adhered  to  while  obtaining  participant  anthropometric  data. 
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References: 


Additional  items  of  environmental  and  performance  measuring 
equipment  are  listed  in  Appendix  D,  These  items  include 
the  G-5  anthropometer , physical  measurement,  motion  pictures, 
sound  tapes,  video  tapes,  photographs  and  event  recorders, 
which  are  also  presented  in  separate  sections  of  this  guide. 

Most  of  the  above  equipment  may  be  obtained  in  kit  form 
for  use  in  HFE  T&E  field  operations.  The  equipment  is 
packaged  into  large  carrying  cases  along  with  the  appro- 
priate cables,  batteries,  operating  manuals,  test  equip- 
ment, and  expendables. 

The  following  two  reports  include  the  use  of  environment 
measuring  equipment: 

a)  Air  Force,  Category  II  System  Development  Test  and 
Evaluation  of  the  4821  Emergency  Mission  Support  System, 
PSTE,  APGC-TR-65-61 , AD  476  172,  Vol.  Ill,  Deputy  for 
Test  Operations,  Air  Proving  Ground  Center,  Eglin  AFB, 
December  1965. 

b)  Martin  Company,  PSTE  Test  Cycle  Report  on  Missile  $M68-11, 
CR-63-43,  AD  405  382,  Denver,  Colorado,  February  1963. 


5.2  System  Measurement 
5.2.1  System  Records  Review 

Description:  There  are  a number  of  typical  test  and  evaluation  program 

records  that  may  be  useful  for  review  by  the  HFE  personnel. 

This  technique,  the  review  of  system  T&E  records,  is  unique 
in  that  there  is  no  direct  contact  between  the  test  evaluator 
and  the  test  participants.  All  that  is  required  on  the  part 
of  HFE  evaluators  is  to  obtain  permission  to  review  the  exist- 
ing test  records  and  to  go  ahead  with  the  tedious  task  of  look- 
ing through  them.  The  evaluator  should,  of  course,  have  some 
sort  of  system  knowledge  to  know  what  he  is  looking  for  in  terms 
of  anticipated  human  performance.  Typically,  system  records  will 
contain  test  logs,  maintenance  records,  and  debriefing  records. 
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The  HFE  evaluator  may  find  data  on  equipment  operation 
problems,  technical  publication  inadequacies,  human 
initiated  errors,  and  training  inadequacies. 


Use/Val idity : 


References: 


Ik. 


This  tecnniquc  is  best  used  for  gathering  man-machine 
performance  data.  Because  the  HFE  does  not  actually  ob- 
serve the  test,  it  is  doubtful  that  sufficient  evaluation 
can  reliably  take  place  by  reading  a word  description  of 
what  occurred.  Human  performance  tests  may  have  to  be  sche- 
duled for  the  purpose  of  formal  observation  by  HFE  personnel. 

The  problem  with  a review  of  test  records  is  that  they  tend 
not  to  be  designed  for  gathering  human  factors  data.  What 
the  HFE  is  able  to  obtai*.  om  these  records  may  be  mislead- 
ing. There  is  significant  risk  that  HFE  problems  that  could 
be  readily  apparent  by  direct  observation,  are  unobserved  or 
obscured  by  other  less  significant  test  data.  In  order  to 
enhance  the  value  of  system  records  review,  the  personnel 
who  initiate  these  records  should  be  indoctrinated  in  the 
value  of  HFE  and  HFE  T&E. 

It  is  generally  agreed  that  the  use  of  this  technique  is 
not  required.  It  is  recommended  that  it  be  performed  only 
when  direct  HFE  observation  is  not  possible.  The  debriefing 
records  should  be  the  most  useful  of  all  the  system  records 
normally  available. 

This  technique  has  been  used  to  a limited  degree  on  the  E-3A 
(AWACS)  program.  The  following  references  include  information 
on  system  records  review  associated  with  other  programs: 

a)  Majesty,  M.  S.,  Human  Factors  Concepts  for  Testing  Complex 
Man/Machine  Systems,  Headquarters  Ballistics  Systems  Div- 
ision, AFSC,  June  1961 . 

b)  Meister,  D. , and  Rabideau,  G.  F.,  Human  Factors  Evaluation 

iji  System  Development,  New  York:  Wiley,  1965. 
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c)  Peters,  G.  A.,  and  Hall,  F.  S. , Missile  System  Safety: 

An  Evaluation  of  System  Test  Data  (Atlas  MA-3  Engine 
System).  ROM  3181-1001,  R-5135,  Reliability  Operations  - 
Human  Factors,  Rocketdyne  Engineering,  March  1963. 

5.2.2  Human  Initiated  Failures 

Description:  A special  form  or  forms  may  be  provided  to  record  only  test 

participant  errors.  In  order  to  do  this  properly,  the  test 
observer  must  be  knowledgeable  of  the  system  procedures  to  the 
extent  that  he  will  notice  operator  or  participant  errors.  The 
form  requires  the  following  information:  where  and  when  the  error 

occurred;  a description  of  exactly  what  the  error  was;  and  estimates 
as  to  the  cause  of  the  error,  both  by  the  observer  and  the  parti- 
cipant. Possible  causes  may  be  presented  in  a short  check  list 
format  (e.g.,  fatigue,  environment,  equioment  design,  etc.) 

Once  this  form  is  completed,  it  should  be  used  to  make  recommenda- 
tions to  design  engineering  or  other  organizations  to  modify  the 
man-machine  interface  design,  or  improve  training,  etc.  The 
error  data  should  also  be  compiled  in  a human  performance  re- 
1 iabi 1 i ty  data  bank. 

Use/Validity:  The  major  problem  with  the  use  of  this  type  of  error  reporting 

technique  is  in  being  at  the  right  place  at  the  right  time  to 
observe  the  error.  There  is  an  extreme  reluctance  or  the  part 
of  participants  to  report  their  own  errors.  In  some  instances, 
observers  are  even  hesitant  to  report  operator  errors.  This  is 
particularly  true  if  the  error  is  quickly  corrected.  The  solu- 
tion to  this  problem  is  proper  indoctrination  and  motivation  to 
the  benefits  of  error  reporting.  Test  observers  must  be  looking 
constantly  for  participant  mistakes  and  must  be  thoroughly 
familiar  with  the  test  participant  (operator)  required  tasks. 

This  technioue  is  particularly  appropriate  to  MIL-H-46855  para- 
graph 3. 2. 4. 3,  "Failure  Analysis",  compliance. 
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This  technique  has  been  used  on  the  Air  Force  E-3A  (AWACS) 
program.  It  is  easy  to  use  in  terms  of  the  time  and  effort 
required  to  complete  the  error  analysis  form.  Additional 
evaluation  of  this  technique  as  compared  to  others  is  in 
Appendix  D.  The  following  Figure  5.2-1  illustrates  a sample 
format  for  the  form. 

References:  Human  Performance  Reliability  Testing  Technique,  Section  5. 1 .6. 

a)  Blanchard,  W.  H. , Recording  Battle  Staff  Performance,  TM- 
2593/000/00,  AD  620  662,  System  Development  Corp. , August 
1965. 

b)  Meister,  D.,  and  Rabideau,  G.  F. , Human  Factors  Evaluation 

in  System  Development,  New  York:  Wiley,  1965. 

c)  Myers,  L.  E. , et  al  Guidebook  for  the  Collection  of  Human 
Factors  Data,  Report  PTB  66-3,  AD  631  023,  HRB-Singer,  Inc., 
State  College,  Pennsylvania,  January  1966. 

5.2.3  Test  Participant  History  Record 

Description:  This  is  not  a direct  test  technique  but  rather  a method  of  im- 

proving the  test  evaluation  process.  The  Test  Participant  History 
Record  form  is  used  to  collect  data  on  personnel  participating 
in  HFE  tests.  This  form  should  be  completed  before  participation 
in  the  tests,  if  possible.  Otherwise,  the  form  may  be  completed 
as  part  of  the  post-test  interview.  The  sample  form  included 
in  the  following  pages  (Figure  5.2-2)  emphasizes  participant 
training,  experience  in  systems  similar  to  the  one  being  tested, 
and  oarticipation  in  previous  testing  related  to  the  same  over- 
all system  presently  being  tested.  This  form  may  need  to  be 
modified  to  suit  the  needs  of  the  particular  test  situation. 

Use/Val idity:  The  purpose  or  use  of  this  form  is  to  assist  in  the  evaluation 

of  the  obtained  test  data.  For  example,  if  the  test  participant 
has  had  little  or  no  experience  in  performing  tasks  similar  to 
the  ones  he  has  been  given  to  do  as  a test  participant,  and  he 
does  very  well,  then  the  conclusion  is  that  the  man-machine 
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OPERATOR  ERROR  ANALYSIS 

INSTRUCTIONS: 

1. 

Z 

z 

4l 

ATTACH  TEST  CONDUCTOR  S OPERATOR  ERROR  LOG  TO  THIS  SHEET. 
TEST  CONDUCTOR  COMPLETE  SECTION  A. 

OPERATOR  COMPLETE  SECTION  B. 

PSTE  REPRESENTATIVE  ATTEND  TWG  REVIEW  AND  COMPLETE  SECTION  C. 

SECTION  A - TEST  CONDUCTOR 

DATE: 

1. 

FLIGHT  NO. 

Z flight  DURATION 

HRS 

_ MIN, 

3. 

FLIGHT  TIME  WHEN  ERROR  OCCURRED 

4. 

DESCRIBE  ERROR:  (WRONG  SWITCH  ACTION,  LATE  SWITCH  ACTION. 

INCORRECT  COMMUNICATION.  ETC.) 

CAUSE  OF  ERROR: 


IF  “UNKNOWN  " REQUEST  OPERATOR  TO  PROVIDE  COMPLETE 
DETAILS  IN  SECTION  Bl. 


SECTION  B - OPERATOR 

PLEASE  STATE  THE  CIRCUMSTANCES/CONDtTlONS  YOU  FEEL  RESULTED  IN 
ABOVE  ERROR  (USE  CONTINUATION  SHEET  IF  NECESSARY). 


2,  WAS  THE  ERROR  DUE  TO  FATIGUE  RESULTING  FROM  THE  TEST  MISSION? 

CHECK:  YES  NO  


SECTION  C - TWG  REVIEW  (CONSULT  WITH  OPERATOR  AND  TEST  CONDUCTOR  AS  NECESSARY) 
1.  ERROR  WAS  DUE  TO: 

ENVIRONMENT  (LIGHTING,  TEMPERATURE,  TURBULENCE,  ALTITUDE,  ETC.) 

EQUIPMENT  DESIGN/FUNCTION/ARRANGEMENT 

SOFTWARE  DESIGN/FUNCTION 

INADEQUATE  PROCEDURE  PRESCRIBED  (INCLUDING  T.O.) 

WRONG  SKILL  FOR  TASK 

OPERATOR  NOT  PROPERLY  TRAINED 

EXCESSIVE  OPERATOR  WORKLOAD 

OTHER  (EXPLAIN  IN  ITEM  2) 

Z ATTACH  A CONTINUATION  SHEET  DESCRIBING  THE  CONTRIBUTING  FACTOR(S)  CHECKED 
ABOVE.  ALONG  WITH  RECOMMENDED  CORRECTION.  IF  ANY. 


Figure  5.2-1,  Human  Initiated  Failure  Form 
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PS  TEST  PARTICIPANT  HISTORY  RECORD 


Figure  5.2-2:  Sample  Test  Participant  History  Record 


interface  being  tested  has  been  well  designed  and  developed. 
If,  on  the  other  hand,  his  performance  is  poor,  the  problem 
may  or  may  not  be  due  to  poor  man-machine  interface  design. 

A more  experienced  test  participant  will  have  to  be  given  the 
same  tasks  to  perform.  The  time  and  effort  it  takes  to 
complete  the  form  is  small,  and  the  potential  value  of  having 
the  test  participant's  significant  history  is  large.  This 
technique  has  been  used  on  the  Minuteman  Personnel  Subsystem 
Test  and  Evaluation  (PSTE)  program. 


References ; 


a)  Air  Force,  Golden  Ram  Personnel  Subsystem  Final  Report, 

Atlas  "D"  Series,  Contract  AF04(647)-619,  September  1961. 

b)  Air  Force,  Personnel  Subsystem  Specification,  SAMSO  Exhibit 
68-19,  Headquarters  Space  and  Missile  Systems  Command,  Sept- 
ember 1968. 

c)  Air  Force,  Test  Plan  for  Category  II  Testing  of  System  416M, 
Prepared  by  416M  Test  Force,  Hanscom  AFB,  November  1963. 

d)  Lindsey,  J.  F. , Bioastronautics  Study  of  B-58  Training, 
Category  III  System  Operational  Test  and  Evaluation , APGC- 
TN-61-8,  AD  259  304,  Bioastronautics  Office,  lir  Proving 
Ground  Center,  Eglin  AFB,  May  1961. 

e)  Martin-Marietta,  Titan  II  Category  II  Observer/Evaluator 
Handbook:  PSTE  Operating  Procedures:  Maintenance,  Logistics, 
etc,  for  SM-68B,  Titan  II,  Denver,  Colorado,  July  1963. 


5.3  Indirect  Manual 
5.3.1  Interviews 

Description:  The  HFE  T&E  interview  technique  is  simply  the  process  of  the  HFE 

test  evaluator  discussing  the  test  events  with  the  test  parti- 
cipants. This  discussion  should  be  structured  in  order  to 
insure  that  the  most  information  is  obtained  in  the  least  amount 
of  time. 

The  first  step  in  the  process  of  conducting  the  interview  is 
to  develop  a format  for  asking  questions  of  the  participants 
( interviev.'ses' . T'^e  format  may  be  structured  like  a checklist 
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to  insure  that  all  pertinent  aspects  of  the  test  are  considered. 
The  second  step  is  to  select  an  interviewer  who  has  had  exper- 
ience with  the  system  being  evaluated.  It  is  important  that  he 
has  observed  the  actual  test  conducted.  The  next  step  is  to 
arrange  a time  to  conduct  the  interview  with  the  test  participant. 

The  interviewee  should  be  questioned  about  the  task  he  has  per- 
formed. He  should  describe  what  he  thinks  his  test  task  con- 
sists of  in  terms  of  his  duties  and  those  of  others.  His  opinions 
should  be  obtained  on  the  adequacy  of  the  equipment,  technical 
data,  logistics  and  preparatory  training. 

The  interview  should  be  conducted  as  soon  as  practical  after 
the  actual  test,  hopefully  within  a few  hours.  If  possible, 
the  interview  should  be  conducted  on  a one  to  one  basis  rather 
than  one  interviewer  questioning  several  participants  at  one 
time.  The  area  selected  for  the  interview  should  be  relatively 
quiet  with  a minimum  of  distractions.  The  time  taken  to  conduct 
the  interview  should  be  less  than  half  an  hour.  Interviews 
which  are  longer  than  this  start  to  get  boring  and  become 
an  imposition  on  the  interviewee. 

The  HFE  interviewer  must  take  care  to  insure  that  he  is  obtaining 
the  interviewees  actual  opinions  as  to  the  test  situations  and 
not  what  the  interviewee  thinks  the  interviewer  wants  to  hear. 

The  participant  must  be  assured  that  he  is  not  being  graded  in 
any  way  on  his  responses.  The  HFE  interviewer  should  try  to 
quickly  develop  a rapport  with  participants . If  the  participant 
agrees,  a tape  recording  may  be  taken  of  the  interview.  However, 
whether  the  participant  agrees  or  not,  some  individuals  tend 
to  be  intimidated  by  the  use  of  tape  recordings  and  caution  must 
be  used  in  this  regard. 

Specific  variations  to  the  general  interview  technique  may  be  of 
use  for  particular  situations.  For  example,  considerable  test 
and  evaluation  data  may  be  obtained  from  training  instructors. 

They  are  particularly  knowledgeable  in  regard  to  student  problems 
with  new  systems  because  of  inadequacies  in  the  system  design. 
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Another  example  of  an  interview  technique  is  the  "critical  in- 


cident technique".  The  critical  incident  technique  consists 
of  a set  of  procedures  for  collecting  observations  of  human 
behavior  in  such  a way  as  to  facilitate  their  potential  use- 
fulness in  solving  practical  problems.  A critical  incident 
is  any  observable  human  activity,  the  purpose  and  effects  of 
which  seems  clear  to  the  observer.  The  five  step  procedure 
is  basically  as  follows:  a)  Determination  of  the  general  pur- 
pose of  the  activity;  b)  Development  of  plans  for  collecting 
incidents  regarding  the  activity  and  instructions  to  the  per- 
sons who  are  to  report  their  observations;  c)  Collection  of 
relative  objective  data;  d)  Analysis  of  the  data;  and  e)  In- 
terpretation and  reporting  of  the  statement  of  the  require- 
ments of  the  activity.  The  gathering  of  the  series  of  inci- 
dents consists  of  inquiry  as  to  most  effective  or  ineffect- 
ive behavior  (or  critical  incident)  of  specified  activities/ 
jobs.  Although  the  incidents  may  be  secured  by  interviews, 
they  may  also  be  obtained  by  written  responses. 

The  end  product  of  the  interview  is  a quantity  of  test  data 
(facts  and  opinions)  to  review  and  evaluate  for  the  purpose 
of  presenting  system  problems  and  recommendations,  and  in 
many  cases  system  verification. 

Use/Validity:  The  interview  is  one  of  the  most  significant  evaluation  methods 

used.  It  is  a simple,  low  cost,  quickly  used  technique.  Every 
test  involves  a certain  amount  of  test  data  that  cannot  be  obtain- 
ed through  normal  observation.  Interviews  with  the  test  partici- 
pants draw  directly  on  this  type  of  data  and  on  the  knowledge  of 
the  presently  available  system  experts.  Interviews  do  not  require 
the  use  of  test  facilities.  They  may  be  conducted  in  an  area 
remote  from  the  test  site. 

The  purpose  of  an  interview  is  to  find  out  either  objective  facts 
related  to  the  system  about  which  the  interviewee  has  some  know- 
ledge, or  subjective  facts,  attitudes,  or  opinions  about  how 
he  feels  about  some  test  aspect.  The  interview  must  be  designed 
to  obtain  these  facts  with  as  much  clarity  and  accuracy  as  possible. 
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References : 


The  interview  attains  its  greatest  value  from  the  relation- 
ship which  is  established  between  the  interviewer  and  the 
respondent.  In  a properly  conducted  interview,  where  a genuine 
rapport  is  established  between  the  interviewer  and  the  inter- 
viewee, it  is  possible  to  obtain  more  detailed  and  reliable 
data  than  from  the  self-administered  questionnaire. 

One  caution  that  must  be  pointed  out  in  the  use  of  interviews 
is  bias  on  the  part  of  the  interviewer  or  interviewee.  Ideally, 
the  interview  results  in  the  interviewee  supplying  accurate  in- 
formation to  the  interviewer.  However,  the  influence  of  bias 
can  alter  the  results  to  such  an  extent  that  the  answers  are  of 
little  or  no  value  in  the  final  analysis.  The  interviewer  may 
bias  the  interview  by  tone  of  voice,  the  way  in  which  the  ques- 
tions are  phrased,  or  even  by  facial  expressions.  These  and 
other  sources  of  bias  can  be  greatly  reduced  through  recognition 
of  the  problem  and  by  training  and  experience. 

Another  caution  associated  with  the  use  of  interviews  is  that 
they  cannot  be  used  as  a substitute  for  direct  test  observation. 

They  should  be  used  as  one  of  several  HFE  test  and  evaluation 
techniques. 

This  technique  has  been  used  by  the  services  and  contractors 
alike  on  numerous  programs  such  as  Minuteman,  E-3A,  WX-50  and 
A7TRAM. 

a)  Air  Force,  PSTE  Standard  Operating  Procedure  (SOP)  for 
Program  279,  Space  Command  and  Control,  SAC,  ATC,  and 
OOAMA,  Bendix,  1963. 

b)  Air  Force,  Pjr^ojinel  Subsystem  Specification,  SAMSO  Exhibit 
68-19,  Headquarters  Space  and  Missile  Systems  Command,  September 
1968. 

c)  Flanagan,  J.  C.,  "The  Critical  Incident  Technique",  Psycho- 
logical Bulletin,  Vol . 51,  No.  4,  July  1954. 
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d)  Inaba,  K. , The  Titan  II  Inertial  Guidance  System  Category 
II  PSTE  and  Maintenance,  Logistics,  Reliability  and  Readi- 
ness (PSTE/MLRR)  Program,  Volume  I,  Serendipity/A. C.  Spark 
Plugs  Coordination  Document  No.  36,  Serendipity  Associates, 
Sherman  Oakes,  California,  September  1963. 

e)  Army,  Man-Material  Systems,  Questionnaire  and  Interview  De- 
sign (Subjective  Testing  Techniques),  TECOM  Pam.  602-1,  Vol . 
1,  Headquarters,  U.W.  Army  TECOM,  July  1975. 

f)  Meister,  D. , and  Rabideau,  G.  F.,  Human  Factors  Evaluation 
in  System  Development,  New  York:  Wiley,  1965. 


Additional  information  on  this  technique  as  compared  with  other 
indirect  observation  techniques  may  be  found  in  Appendix  D. 

5,3.2  Questionnaires 

Description:  The  basic  tool  for  obtaining  subjective  data  is  the  questionnaire 

It  is  the  most  frequently  used  and  most  difficult  to  construct 
of  the  subjective  techniques.  The  questionnaire  provides  a 
structured  method  for  asking  a series  of  predetermined  questions 
in  order  to  obtain  measurable  expressions  of  attitudes,  prefer- 
ences and  opinions.  The  design  of  a questionnaire  which  will 
produce  valid  and  reliable  results  requires  a measure  of  skill 
and  experience.  Unfortunately,  questionnaire  design  and  con- 
struction cannot  be  taught  from  books;  the  requirements  for  each 
test  are  somewhat  different  and  present  new  and  different  problem 
However,  there  are  certain  rules  and  principles  of  questionnaire 
design  and  administration  which,  when  followed,  eliminate  some 
of  the  more  common  oitfalls  which  result  in  faulty  questions 
and  invalid  results.  The  following  material,  especially  the 
references,  are  intended  to  provide  guidance  for  planning, 
designing  and  administering  the  questionnaire. 


77 


The  method  of  questionnaire  design  applicable  to  the  types 
of  tests  conducted  by  HFE  T&E  personnel  may  be  divided  into 
seven  logical  steps; 

a)  Preliminary  planning. 

b)  Selection  of  the  question  form. 

c)  Wording  of  the  questions. 

d)  Formulating  the  questionnaire. 

e)  Pretesting. 

f)  Administering  the  questionnaire. 

g)  Quantification  and  analysis  of  questionnaire  data. 

The  preparation  of  a questionnaire  requires  great  care  and  a 
background  knowledge  of  the  system  to  be  tested.  Knowledge  also 
is  required  regarding  the  background  of  personnel  to  whom  the 
questionnaire  will  be  administered,  and  the  type  of  analysis 
which  will  be  made  of  the  results.  Too  often  a questionnaire 
is  prepared  with  insufficient  planning.  The  problems  involved 
and  the  weaknesses  in  the  design  are  frequently  not  recognized 
until  such  time  as  the  results  are  interpreted. 

There  are  four  basic  question  forms  that  may  be  used  in  a 
questionnai re: 

a)  The  open-end  or  free-answer. 

b)  The  dichotomous  or  two-way. 

c)  The  multiple  choice 

d)  The  rating  scale. 

Each  form  has  its  merits  and  disadvantages  of  which  the  ques- 
tionnaire designer  must  be  aware  and  must  weigh  carefully  before 
final  selection.  No  one  question  form  is  superior  to  the  others 
in  all  cases.  In  order  to  select  one  form  over  another,  the 
designer  must  be  aware  of  the  advantages  and  disadvantages  of 
each  and  choose  that  form  which  best  meets  the  needs  of  the 
particular  test  situation. 
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The  most  important,  and  also  the  most  difficult,  aspect  of 
questionnaire  construction  is  the  wording  of  the  questions. 

Most  authorities  agree  that  faulty  or  improper  wording  of 
questions  accounts  for  the  greatest  source  of  error  in  the 
questionnaire  technique.  Errors  and  distortions  in  the  final 
data  are  often  caused  by  misunderstanding  and  misinterpretation 
of  questions  due  to  use  of  an  improper  vocabulary  level  and 
ambiguous  phrasing.  In  addition  to  selecting  the  question 
forms  and  wording  the  questions,  it  also  is  necessary  to  con- 
sider such  factors  as  the  sequence  of  the  questions  and  the 
format  for  presentation  and  data  collection.  A check  must 
be  made  of  all  questions  to  insure  complete  and  accurate  cover- 
age of  all  data  required  by  the  test  objectives  and  test 
critical  issues. 

A questionnaire  is  subject  to  many  variables  and  must  not  be 
assumed  to  be  perfected  until  it  has  been  subjected  to  trial 
use.  The  pretest  provides  an  opportunity  to  try  the  questionnaire 
out  on  a small  sample  of  respondents.  The  results  of  this  trial 
may  then  be  used  to  make  revisions  and  improvements  as  necessary 
before  test  administration.  The  pretest  is  the  final  and  vali- 
dating step  in  the  method  of  questionnaire  construction. 

The  product  obtained  from  administration  of  the  questionnaire 
consists  of  subjective  words  or  ohrases.  This  information  may 
be  quantified  and  converted  to  figures  or  numbers  that  can  be 
tabulated  and  analyzed.  The  end  product  of  the  questionnaire 
may  be  a simple  frequency  distribution  of  responses  to  each 
question  summarized  in  terms  of  numbers,  proportions  or  per- 
centages. The  data  may  be  further  summarized  to  include  averages, 
standard  deviations,  or  correlations.  The  summaries  also  may  in- 
clude statistical  analyses  showing  the  statistical  significance 
of  differences  or  correlations  obtained.  These  quantified  data 
must  then  be  tabulated  and  analyzed.  The  results  usually  are 
summarized  in  tabular  form  for  inclusion  in  a final  report. 
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When  compared  to  the  interview,  there  are  several  simi- 
larities and  differences  with  the  questionnaire.  Both  the 
questionnaire  and  interview  should  be  conducted  within  a few 
hours  of  the  test  for  best  results.  Both  techniques  may  be 
conducted  away  from  the  test  area.  Although  the  Questionnaire 
must  be  more  structured  than  the  interview,  the  questionnaire 
may  still  include  open-ended  questions.  The  differences  are 
in  that  HFE  personnel  need  not  be  present  while  the  questionnaire 
is  being  filled  out.  The  questionnaire  is  inherently  easier  to 
use  in  evaluation  or  analysis  of  the  participant  responses. 

Use/Val idity:  The  questionnaire  is  a subjective  measurement  tool  for  systematically 

obtaining  attitudinal  responses  from  a selected  group  of  individuals. 
The  function  of  the  questionnaire  is  to  communicate  information. 

When  properly  formatted,  it  also  aids  in  the  tabulation  of  data 
and  analysis  of  results.  The  questionnaire  is  used  to  assess  a 
wide  variety  of  qualitative  variables  such  as  acceptance,  ease 
of  use  and  preference.  It  may  be  administered  to  small  groups  of 
technical  personnel,  such  as  those  involved  in  highly  controlled 
engineering  tests,  or  to  larger  representative  cross-sections  of 
service  personnel. 

Knowledge  of  individual  or  group  attitudes  provides  valuable  in- 
formation regarding  reactions,  feelings,  and  preferences  toward 
military  systems.  Since  attitudes  determine  behavior,  questionnaire 
responses  of  a representative  sample  of  the  population  permit  a 
reliable  estimate  of  group  reactions  to  systems  in  actual  use. 

These  results  also  may  be  used  to  anticipate  and  thereby  avoid 
future  developmental  problems. 

The  questionnaire  is  appropriate  for  use  in  all  types  of  tests. 

It  should  be  used  to  obtain  subjective  data  when  objective  measure- 
ment is  not  feasible  and  when  qualitative  data  are  needed  to  supple- 
ment objective  measurements.  However,  it  should  not  be  used  in 
place  of  direct  observation  techniques  if  observation  is  possible. 


80 


A disadvantage  of  the  questionnaire  is  that  test  participants 
won't  respond  in  writing  to  the  degree  that  they  would  in  talk- 
ing in  a response  to  an  interview.  The  effort  to  write  respon- 
ses to  open-ended  questions  is  greater  than  the  effort  to  talk. 
Another  diSi^dvantage  of  the  questionnaire,  compared  to  the  inter- 
view, is  the  inability  of  the  HFE  observer  to  pursue  a partici- 
pant response  that  is  unexpected  but  potentially  fruitful. 

One  of  the  most  difficult  problems  to  overcome  in  questionnaire 
design  is  the  misunderstanding  on  the  part  of  individuals  as  to 
what  a questionnaire  is  and  how  it  should  be  used.  There  are 
those  who  believe  that  anyone  who  can  write  well  and  use  a little 
common  sense  can  construct  a good  questionnaire.  The  seriousness 
of  this  faulty  assumption  is  illustrated  by  the  fact  that  an  im- 
properly designed  and  poorly  worded  questionnaire  will  still 
yield  data  in  the  form  of  numbers,  frequencies  and  percentages. 

These  numbers  are  amenable  to  statistical  analysis  and  may  even 
produce  statistically  significant  findings.  The  real  tragedy  is 
that  these  erroneous  findings  may  be  used  to  draw  false  conclu- 
sions which,  in  turn,  contribute  to  faulty  critical  decisions 
regarding  the  utility  of  an  item. 

References : Functional  description  inventories  (FDI's)  which  are  discussed  in 

the  following  section. 

a)  Air  Force,  Test  Plan  for  Category  II  Testing  of  System  482L, 
AN/TSQ-47  Emergency  Mission  Support  System,  Air  Proving 
Ground  Center  (AFSC),  Eglin  AFB,  September  1962. 

b)  Army,  Man-Material  Systems,  Questionnaire  and  Interview  Design, 
(Subjective  Testing  Techniques),  TECOM  Pam  602-1  , Vol . 1 , 
Headquarters,  U.S.  Army  TECOM,  July  1975. 

c)  Hattendorf,  G.  A.,  et  al , C-141A  Category  II  Systems  Evaluation, 
Technical  Report  65-38,  Air  Force  Flight  Test  Center,  Edwards 
AFB,  July  1966. 

d)  Myers,  L.  B.,  et  al  , Guidebook  for  the  Collection  of  Human 
Factors  Data.  Report  PTB  66-3,  AD  631  023,  HRB-Sinqer,  Inc., 
State  College,  Pennsyl vania , January  1966. 
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e)  Potempa,  K.  W.,  "Open-Ended  Maintenance  Questionnaire", 
A Catalog  of  Human  Factors  Techniques  for  Testing  New 
Systems , AFHRL-TE-68-5,  February  1969. 


Additional  information  on  this  technique  as  compared  with 
other  indirect  observation  techniques  may  be  found  in 
Appendix  D. 

5.3.3  Functional  Description  Inventory  (FDI) 

Description:  The  FDI  is  a new  HFE  T&E  assessment  methodology  or  special 

case  questionnaire.  The  FDI  requires  a series  of  investiga- 
tions analyzing  the  operational  functions  of  crewmembers,  with 
an  essential  part  involving  the  determination  of  roles,  duties 
and  tasks  performed  by  each  crewmember.  Subsequent  to  this, 
analysis  time  is  provided  in  order  that  the  crewmembers  can 
judge  how  important  these  roles,  duties  and  tasks  were  for 
mission  success.  They  also  judge  how  frequent  they  are  per- 
^ formed  on  a typical  mission,  how  adequate  the  training  has  been 

to  insure  effective  performance  of  the  task,  and  finally,  how 
effective  the  particular  system  has  been  in  accomplishing  these 
operational  functions. 

Once  the  approval  has  been  given  to  use  the  FDI,  the  procedural 
steps  to  be  followed  in  its  development  are  as  follows: 

a)  Source  material  is  gathered  to  provide  guidelines  for 
selection  and  working  of  duties  and  tasks. 

b)  Local  technical  advisors  (crewmembers  experienced  in  the 
particular  position  under  study)  review  pools  of  duty  and 
task  items  for  applicability.  They  also  develop  additional 

I items  as  required. 

c)  A preliminary  FDI  is  developed  based  on  reviewer  comments, 

I recommendations  and  additions  to  the  duty/task  list. 

\ d)  On  the  bas’s  of  this  review,  a final  FDI  is  developed. 

i 

1 
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Use/Validity: 


[ 


f 

i 

I 

References: 


e)  The  FDI  is  distributed  to  test  participants  (experienced 
crewmen)  who  then  fill  them  out. 

f)  Statistical  computations  are  generated  by  computer  programs. 
For  example,  items  of  statistical  interest  are  standard 
deviations  as  to  question  responses  in  order  to  determine 
later  agreement. 

f)  System  problems,  as  indicated  by  the  consensus  of  the  crew, 
are  noted  for  feedback  to  the  existing  system  development 
procedures. 

The  FDI  has  been  developed  in  detail  by  the  Naval  Air  Test 
Center  at  Patuxent  River.  The  P-3C,  E-2C  and  S-3A  have  been 
evaluated  with  this  technique.  There  are  several  advantages 
to  its  use.  For  example,  it  may  be  used  as  a checklist  to 
cover  all  aspects  of  the  system  test.  The  data  obtained  in 
this  technique  comes  from  system  experts.  In  turn,  the  system 
expei^ts  are  trained  as  to  what  HFE  criteria  are  significant. 

The  technique  may  be  considered  extremely  thorough. 

The  disadvantages  in  using  this  technique  are  the  time  and 
effort  required  to  develop  the  questionnaires.  They  must  be 
unique  to  each  program  being  tested.  The  time  and  effort  for 
the  crewmen  to  complete  the  questionnaires  are  also  considerable. 
Of  much  less  consequence,  but  still  to  be  considered,  is  the 
problem  of  obtaining  permission  to  use  crewmen's  time,  first  to 
develop  the  questionnaires,  and  second  to  complete  the 
questionnaires. 

Although  not  as  complete  as  the  FDI,  other  more  generalized 
questionnaires  could  be  used  in  place  of  it. 

Helm,  W.  R.,  Fourth  Interim  Report  - Function  Description 
Inventory  as  a Human  Factors  Test  and  Evaluation  Tool:  An 

Empirical  Validation  Study,  SY-127R-76,  Naval  Air  Test  Center, 
July  1976. 

Additional  data  on  this  technique  as  compared  with  other  in- 
direct observation  techniques  may  be  found  in  Appendix  D. 


83 


Description:  The  Personnel  Activity  Analysis  Radio  System  is  essentially 

a direct  observation  questionnaire  technique.  It  is  based 
on  the  use  of  2-way  radios  between  one  HF  engineer  and  sev- 
eral maintenance  technicians  or  crews.  It  is  not  intended 
to  be  used  as  an  observation  technique  for  formal  HFE  test  and 
evaluation.  It  is  intended  for  quick  and  efficient  gathering 
of  human  factors  data  from  maintenance  operations  being  per- 
formed on  certain  systems  in  the  operational  inventory.  Tech- 
nicians are  required  to  report  in  from  the  location  of  their 
work.  They  are  asked  questions  in  regard  to  equipment  problems, 
procedural  problems,  delays  and  potential  hazards. 


The  technique  requires  the  use  of  an  off-the-shelf  base  station, 
receiver- transmi tter  with  encoder  unit,  several  portable  re- 
ceiver-transmitters, and  a battery  charger.  The  portable  unit 
should  weigh  less  than  a few  pounds  and  be  less  than  50  cubic 
inches  in  size.  The  portable  unit  antenna  should  not  require 
adjustments  and  should  not  interfere  with  work  activities.  A 
concealed  or  self-contained  system  is  desired.  The  portable 
units  should  also  have  a hands-off  capability  for  talk-listen 
after  the  units  are  switched  on.  The  base  station  should  have 
a capability  to  make  a hard-line  connection  between  transmitter- 
receiver  and  a tape  recorder  to  permit  direct  recording  of  in- 
coming and  outgoing  communications. 


An  extremely  significant  requirement  for  the  use  of  this  system 
is  the  permission  from  security  and  maintenance  control  personnel. 


Use/Validi ty:  The  results  from  use  of  the  PAARS  are  in  the  data  gathered  for 

operational  evaluation.  Maintenance  crew  task  performance,  in 
terms  of  time  required  and  errors  made,  are  compiled.  There  is 
a significant  payoff  potential  in  correlating  this  operational 
data  to  formal  HFE  test  and  evaluation  data. 
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Although  this  technique  has  been  used  successfully,  it  is 
not  significantly  superior  to  the  more  standard  direct  inter- 
view or  questionnaire  techniques.  The  PAARS  is  probably  best 
for  determining  equipment  and  tool  problems.  It  obtains  an 
objective  response  to  questions  while  the  data  is  still  fresh 
in  the  mind  of  the  technician.  The  technique  saves  travel  time 
for  th€  HFE  observer  or  base  station  operator. 

The  disadvantages  to  the  use  of  this  technique  are  in: 

a)  the  HFE  observer  cannot  see  the  problem  or  equioment  for 
himself, 

b)  the  technique  is  limited  to  use  with  ground  based  aircraft 
maintenance  task  tests, 

c)  the  technicians  must  remember  to  use  it  or  be  reminded  by 
the  HFE, 

d)  it  requires  the  full  time  of  an  HFE  to  monitor  the  base 
station,  and 

e)  the  technique  is  limited  in  use  for  gathering  test  data 
alone.  It  is  not  useful  for  evaluation  of  test  data. 

This  technique  was  used  on  the  407L  communications  system  and 
for  B-52  flight  line  maintenance. 

References : Askren,  W.  B. , et  al , A Voice-Radio  Method  for  Collecting  Human 

Factors  Data,  AFHRL-TR-68-1 0,  Air  Force  Systems  Command,  Wright- 
Patterson,  January  1969. 

Additional  data  on  this  technique  as  compared  with  other  indirect 
observation  techniques  may  be  found  in  Appendix  D. 

5.3.5  Cooper-Harper  Scale 

Description:  The  Cooper-Harper  Scale  is  a well  known  technique  used  to  evaluate 

aircraft  handling  qualities  through  the  use  of  pilot  ratings.  The 
pilot  participants  are  requested  to  provide  their  evaluation  of 
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given  flight  tasks,  subphases,  phases,  or  even  the  total 
flight  (or  mission).  As  can  be  seen  from  examination  of 
Figure  5.3-1,  which  is  the  Cooper-Harper  aircraft  handling 
qualities  rating  scale,  the  technique  is  basically  dichotomous. 

The  pilot  decides  if  the  aircraft  is  controllable  for  a given 
task,  sub-phase,  etc.  If  the  aircraft  is  controllable  he  decides 
if  the  mission,  ohase,  etc.,  performance  is  adequate  with  a 
tolerable  workload.  If  not  adequate,  he  chooses  (on  a scale 
of  three)  how  bad  the  major  deficiencies  are.  If  adequate, 
he  determines  if  the  performance  is  satisfactory  without  the 
need  for  improvement.  If  not  satisfactory,  he  determines  (on  a 
scale  of  three)  how  bad  the  deficiencies  are.  If  satisfactory , 
he  determines  (on  a scale  of  three)  how  satisfactory  the  per- 
formance is. 

As  can  be  seen  from  the  figure,  the  descriptions  of  aircraft 
handling  performance  are  both  clearly  and  briefly  defined. 
Reference  to  pilot  skill  level  has  been  deleted  from  a previous 
version  of  the  Cooper-Harper  Scale.  Pilot  judgments  as  to  the 
handling  qualities  are  to  be  made  on  the  basis  of  their  own 
skill.  Generally,  the  pilots  requested  to  use  this  technique 
are  quite  skilled,  but  any  variations  in  pilot  skill  should 
average  out  by  using  large  populations  of  pilot  test  oarticipants . 

The  qualitative  statements  as  to  handling  qualities  are  converted 
into  "quantitative"  numbers  only  for  purpose  of  convenience.  The 
scale  of  ten  is  a result  of  ten  categories  of  evaluation.  The 
scale  is  ordinal  as  opposed  to  eoual  interval.  The  use  of  evalua- 
tions halfway  between  those  indicated  are  acceptable  but  not 
encouraged  (e.g.,  ratings  of  1 and  1/2,  2 and  1/2,  3 and  1/2,.  . . 
9 and  1/2). 

In  addition  to  obtaining  the  above  Cooper-Harper  rating  scale 
data,  the  HFE  evaluator  is  encouraged  to  obtain  the  pilots' 
comments  by  way  of  a questionnaire  or  interview. 
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Figure  5.3-1:  The  Revised  Cooper-Harper  Scale  (Handling  Qualities  Rating  Scale) 


L 


Use/Validity:  The  pilot  test  particpant's  subjective  opinion  tends  to  be 

the  most  important  evaluation  item  of  the  air  vehicle.  Un- 
fortunately, the  correlation  between  pilot  ratings,  pilot  be- 
havior, and  vehicle  characteristics  are,  at  best,  highly  quali- 
tative. This  situation  has  not  improved  as  vehicles  and  assoc- 
iated pilot/vehicle  handling  qualities  considerations  have 
steadily  increased  in  complexity. 

A series  of  evaluation  techniques  was  developed  in  an  attempt 
to  properly  determine  aircraft  handling  qualities.  The  first 
in  this  series  was  the  Cooper  evaluation  scale  developed  by 
G.  E.  Cooper  and  NASA.  Cornell  Aeronautical  Laboratory  (R.  P. 
Harper)  then  developed  its  own  rating  scale.  Because  of  in- 
adequacies of  both  of  these  rating  techniques,  they  were  wisely 
combined  into  a new  scale  and  then  revised  again  to  the  scale 
shown  in  Figure  5.3-1.  This  rating  device  is  now  widely  used 
because  it  is  relatively  cost  effective,  simple  to  use,  and 
more  accurate  than  other  similar  HFE  evaluation  techniques. 

The  source  of  the  data  comes  directly  from  the  user  experts, 
the  pilots. 

The  difficulties  experienced  with  the  use  of  this  scale  are 
listed  below: 

a)  The  scale  category  descriptors  may  be  somewhat  incomplete 
or  ambiguous  for  some  users. 

b)  The  qualitative  nature  of  the  scale:  There  is  difficulty 

in  relating  subjective  measures  with  vehicle  and  operator 
parameters.  Pilots  may  be  unable  to  articulate  the  primary 
cause  of  a problem. 

c)  The  scales  are  ordinal , and  of  such  nature  as  to  have 
practically  no  choice  of  having  equal  intervals  on  some 
hypothetical  underlying  interval  scale. 
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d)  The  scales  are  subject  to  misuse  by  design  engineers. 

e)  The  scales  are  useful  for  flight  handling  qualities  only. 

f)  It  is  difficult  to  deal  with  parameters  such  as  tran- 
sient disturbances  (e.g.,  windshear  or  subsystem  failure) 
or  pilot  stress. 

If  the  Cooper-Harper  scale  is  not  used  for  flight  handling 
quality  evaluation,  the  test  evaluator  must  rely  on  air- 
craft installed  sensor  data  (e.g.,  accelerometers)  and  on 
pilot  interviews  and  questionnaires. 

References:  a)  Cooper,  G.  E. , and  Harper,  R.  P. , The  Use  of  Pilot  Rating 

in  the  Evaluation  of  Aircraft  Handling  Qualities,  NASA  TN 
D-5153,  N69-22539,  April  1969. 

b)  McDonnel , J.  D.,  Pilot  Rating  Techniques  for  the  Estimation 
and  Evaluation  of  Handling  Qualities.  System  Technology,  Inc., 

Hawthorne,  California,  AFFDL-TR-68-76,  AD-681845,  December 
1968. 

Additional  data  on  this  technique  as  compare.  *h  other  indirect 
observation  techniques  may  be  found  in  Apr  j. 

i 

5.3.6  Problem  Incident  Report  | 

Description:  This  form  is  used  whenever  the  data  source  indicates  an  unsatis- 

factory condition  relating  to  the  operator/maintainer  element  of 
the  system  being  tested.  The  terminology  for  this  technique  or  i 

form  varies.  It  is  often  called  an  "Unplanned  Event  Record", 

"Crew  Interface  Data",  or  just  a "Squawk  Sheet".  In  many  instances, 
this  report  is  completed  by  the  HFE  test  observer.  However,  it 
may  be  completed  by  anyone  observing  HFE  related  problems.  If 
completed  by  other  than  HFE  personnel,  an  interview  with  the 
originator  is  recommended.  Frequently,  the  item  can  be  closed 
out  as  the  result  of  the  interview.  Numbers  are  assigned  to 
each  new  Problem  Incident  Report.  Records  are  kept  on  each 
report  as  they  are  acted  upon  by  HFE  personnel  to  achieve 
problem  resolution.  As  a matter  of  interest  and  motivation, 
feedback  is  provided  the  incident  report  originators. 
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Use/Validity: 


References; 


In  the  "initiation  of  follow-up  action,  there  are  at  least 
three  paths  to  follow: 

a)  If  the  evaluation  determines  that  no  further  action 

is  indicated,  the  rationale  for  closing  out  the  problem 
is  provided  and  the  report  is  closed  out. 

b)  The  HFE  test  group  will  make  recommendations  as  to 
design,  technical  publication,  maintainability,  training, 
etc.,  as  to  solutions  to  the  problems. 

c)  If  the  problem  is  sufficiently  serious,  additional  organ- 
izations besides  the  HFE  test  grouo  are  notified.  They 
may  wish  to  take  the  lead  in  solving  the  problem. 

d)  The  (customer)  system  program  office  may  also  be  contacted 
if  the  problem  is  a serious  one. 

This  technique  is  easy  to  use  both  in  terms  of  effort  and  time. 

If  well  managed,  it  allows  other  groups  (observers  and  parti- 
cipants) to  provide  HFE  test  data  at  any  time.  The  system 
experts  may  be  the  data  source.  The  only  possible  criticism 
of  this  technique  could  be  said  about  any  other;  it  is  not 
sufficient  to  use  this  technique  alone  to  do  a proper  HFE 
test  job.  If  this  technique  is  not  used,  there  will  be  more 
of  a burden  or  obligation  on  the  part  of  the  HFE  personnel 
to  find  all  the  HFE  related  system  problems. 

This  technique  has  been  used  on  the  Air  Force  E-3A  (AWACS) 
program.  Additional  evaluation  of  this  techniaue  when 
compared  with  others  is  in  Appendix  D.  The  following  Figure 
5.3-2  illustrates  a sample  format  for  the  form. 

a)  Meister,  D. , and  Rabideau,  G.  F.,  Human  Factors  Evaluation 
in  System  Development,  New  York;  Wiley,  1965. 

b)  Peters,  G.  A.,  and  Hall,  F.  S. , Missile  System  Safety:  An 

Evaluation  of  System  Test  Data  (Atlas  MA-3  Engine  System). 

ROM  3181-1001,  R-5135,  Reliability  Operations  - Human  Factors, 
Rocketdyne  Engineering,  March  1963. 

Additional  information  on  this  technique  as  compared  with  other  in- 
direct observation  techniques  may  be  found  in  Appendix  D. 
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ITEST  SHEET  TITLE 


REPORT  NO  lOATE 


■ RELATED  REPORT 


VEHICLE  TYPE  I VEHICLE  SERIAL  NO(SI  I TEST  LOCATION 


■ MAJOR  SYSTEM 


I SUBSYSTEM 


I COMPONENT  PART  NO/SERIAL  NO 


CIO  IDENTIFICATION 


TYPE  HEQ  BIOMED/LS  n PP  D TRAINING  Q TECH.  PUBS.  Q SAFETY  Q mQ 

DEFICIENCY  CIRCUMSTANCES/DESCRIPTION/CAUSES  (CONTINUE  ON  SEPARATE  PAGE  IF  NECESSARY) 


FLOCAL  ACTION 


r^COMMENOATION 


HAZARD  CODE 
(MIL-ST0882) 


HAZARD  CODE.  EFFECT  AND  IMPACT  SUMMARY 
POTENTIAL  HAZARD  EFFECT  I MISSION  IMPACT 


POTENTIAL  HAZARD  EFFECT  MISSION  IMPACT  CORRECTION 

CATEGORY 

□ none  □ delays  □ maintenance 

□ MAJOR/MINOR  DAMAGE  DSUBSYSTEM  GRESTRICTS  GCREW  EFFECTIVENESS  GmANDATORY 

□ MAJOR/MINOB  INJURY  Q PERSONNEL  C DEGRADES  GSYSTEM  PERFORMANCE  GDESIRABLE 

□ loss  □ VEHICLE  □ PREVENTS  GMISSION 


HAZARD  CODE  AMPLIFICATION 

CONTACT 

ORGANIZATION 

M/S 

PHONE 

DISPOSITION 

Ii^aVE 

Figure  5.3-2,  Sample  Problem  Incident  Report 
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5.4  Automatic  Recording 
5.4.1  Motion  Pictures 

Description;  This  technique  is  similar  to  the  use  of  video  tapes.  It  is  the 
process  of  filming  participant  performance  as  a part  of  a system 
test. 

As  with  video  tapes,  actual  prototype  hardware  or  sophisticated 
mockups  should  be  available  to  justify  the  use  of  this  technique. 
Less  sophisticated  mockups  imply  more  uncertainty  in  design, 
and  therefore  a greater  risk  in  expending  a motion  picture  effort 
on  unsuccessful  concepts. 

Trained  test  participants  must  be  available  for  observation  of 
their  appropriate  tasks.  The  cameraman,  and  particularly  the 
HFE  observer,  should  be  familiar  with  the  test  operation  being 
performed.  The  knowledge  of  when  to  take  close-in  footage 
of  a particular  critical  task  is  important.  As  in  the  case 
with  video  cameras,  a dry  run  is  recommended  to  insure  the 
filming  is  properly  performed.  Consultation  with  all  personnel 
familiar  with  the  anticipated  test  events  is  advised. 

The  following  equipment  is  necessary  to  implement  this  technique: 

a)  camera  and  (film) 

b)  lens 

c)  lights 

d)  projector 

e)  screen 

A tripod  may  be  required,  depending  on  the  test  situation. 

Permission  to  use  cameras  in  secure  areas  must  be  obtained  and 
the  camera  equipment  and  cameraman  properly  scheduled. 

Use/Val idity:  This  technique  was  comparatively  more  useful  before  the  de- 

velooment  of  video  tapes.  Video  tapes  are  now  becoming  more 
Dopular  for  that  type  of  test  and  evaluation  process. 
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References : 


However,  when  compared  to  all  other  techniques,  motion  pictures 
still  offer  the  advantages  of:  permanent  precise  records  of 

observed  events,  repeated  observations  of  the  same  event,  slow 
and  fast  motion  study  of  real-time  events,  use  in  hazardous 
areas,  and  record  of  task  activities  as  well  as  the  related 
background  situation.  The  data  gathered  may  be  presented  to 
large  groups  of  people. 

The  disadvantages  are  in  the  cost  and  effort  to  provide  the 
proper  equipment,  particularly  for  processing  and  viewing  the 
film.  Skilled  technicians  are  generally  required  for  the 
filming  of  motion  pictures. 

Motion  pictures  are  not  as  useful  as  video  tapes  in  that  they 
must  be  processed  to  be  viewed.  Instant  playback  of  a film 
cannot  be  made  to  insure  the  adequacy  of  that  particular  test 
record.  After  the  processing,  a projector  and  screen  are 
required.  The  film  cannot  be  reused  as  video  tape  can.  How- 
ever, the  cost  of  the  least  expensive  movie  equipment  is  less 
than  the  least  expensive  video  equipment.  The  process  of 
recording  and  presenting  observed  test  tasks  in  slow  motion 
or  fast-action  is  cheaper  with  motion  pictures. 

Video  tapes. 

The  following  two  reports  include  information  on  the  use  of 
motion  pictures  in  HFE  T&E; 

a)  Adams,  J.  A.  and  McAbee,  W.  H.,  A Program  for  Evaluation 
of  Human  Factors  in  Category  II  Testing  of  Air  Weapons 
Control  System  4121  (Phase  II  Configuration),  PGN  Document 
62-1,  Deputy  for  Bioastronautics,  Air  Proving  Ground  Center, 
Eglin  AFB,  May  1962. 

b)  Majesty,  M.  S. , Human  Factors  Concepts  for  Testing  Complex 
Man/Machine  Systems,  Headquarters  Ballistic  Systems  Division, 
AFSC,  June  1 961 . 

Additional  information  on  the  technique  as  compared  with  other 
automatic  recording  techniques  may  be  found  in  Appendix  D. 
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5.4.2  Sound  Tapes 

Description:  The  use  of  this  technique  is  now  so  common  that  a description 

is  somewhat  superfluous.  Tape  recorders  are  now  both  inex- 
pensive and  portable.  They  are  used  extensively  for  tasks  other 
than  formal  test  observation.  Their  use  in  HFE  T&F.  is  somewhat 
like  that  of  video  tapes  but  without  the  restrictions  of  size, 
security,  transportation  and  cost. 

Test  observers  commonly  use  sound  tape  recorders  to  maintain  a 
complete  record  of  test  conversation  and  events.  Test  notes  may 
be  verbally  entered  by  the  observers  themselves.  The  recorders 
may  also  be  used  to  record  participant  interview  comments.  The 
recorder  may  be  linked  into  the  intercommunication  system  if  such 
is  used  as  a part  of  a large  scale  multioperator  test.  The  use 
of  both  sound  tapes  and  video  tapes  together  is  frequently  valuable. 

Use/Val i di ty : Sound  tapes  are  now  a well  used  test/evaluation  technique.  Their 

use  is  extremely  easy  and  inexpensive.  They  have  the  same  ad- 
vantages as  the  video  tapes  in  that  they  are  a permanent  record 
of  events  (audio),  they  may  be  repeatedly  reviewed,  they  may  be 
used  in  hazardous  areas,  they  require  no  time  to  process,  and 
can  be  used  with  time  tags  if  desired.  In  addition  to  this, 
sound  tape  recordings  negate  the  need  for  detailed  handwritten 
notes. 

One  disadvantage  to  the  use  of  the  recordings  is  in  the  quality 
of  the  reproduction  if  a high  ambient  noise  is  present  near  the 
test  data  being  recorded.  Another  possible  disadvantage  is  if 
the  test  participant  becomes  self-conscious  due  to  the  use  of 
the  recorder.  This  would  be  more  noticeable  during  an  interview. 

I If  the  tape  recorders  are  not  used,  good  note  taking  becomes 

I much  more  important. 
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Additional  information  on  this  technique  as  compared  to  other 
automatic  recording  techniques  may  be  found  in  Appendix  D. 

5.4,3  Video  T apes 

Description:  This  test  and  evaluation  technique  is  the  use  of  video  cameras 

and  related  equipment  to  make  video  tape  recordings  for  detailed 
review  and  evaluation  of  operator  and  maintenance  personnel  tasks. 

Actual  prototype  hardware  or  extremely  sophisticated  mockups  should 
be  available  to  justify  the  use  of  this  technique.  Trained  test 
participants  must  be  available  for  HFE  evaluator  observation  of 
ir  appropriate  tasks.  The  camera  operator(s)  and  particularly 
HFE  evaluator  coordinating  the  video  data  recording  should  be 
jubonably  familiar  with  the  test  operation  being  performed.  The 
knowledge  of  when  to  use  the  zoom  lens  to  home  in  on  a particular 
critical  task  is  important.  In  order  to  be  sure  all  the  more  crit- 
ical tasks  are  properly  recorded,  dry  (or  test)  runs  of  the  test 
may  be  advisable.  Consultation  with  all  personnel  familiar  with 
the  anticipated  test  event  is  reconiiiended. 

The  following  equipment  is  necessary  to  implement  this  technique: 

1)  video  tape  recorder 

2)  camera  (preferably  portable) 

3)  zoom  lens 

4)  monitor 

5)  lights 

Additional  lenses,  monitors  and  tripods  may  be  desired  depending 
on  the  complexity  of  the  test.  Sound  recording  equipment  may 
also  be  desired.  There  are  a number  of  easy-to-use  video  tape 
recording  systems  which  might  be  made  available  to  HFE  personnel 
at  the  test  sites  and  at  contractor  facilities. 
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Problems  associated  with  the  use  of  video  recordings  involve: 
the  logistics  of  transporting  the  equipment  to  the  test  site; 
the  security  of  the  equipment;  permission  to  record  any  occur- 
rences in  secure  areas  (e.g.,  restricted  flight  line  areas); 
scheduling  of  the  video  equipment  and  a cameraman;  and  request 
to  perform  recording  on  a possible  test  interference  basis. 

Use/Validity:  There  is  little  doubt  that  given  the  video  tapes  and  proper 

display  equipment,  the  use  of  this  technique  is  of  notable 
value.  However,  the  cost  effectiveness  of  the  technique  must 
be  considered  to  be  dependent  upon  the  complexity  of  the  task 
needing  evaluation.  Possible  transportation  and  lighting 
problems  should  be  considered  also  before  commitment  to  the 
use  of  this  technique. 

Careful  review  of  tape  playbacks  can  reveal  human  errors  and 
excessive  task  times  not  previously  capable  of  being  detected. 

The  application  of  maintenance  crew  teamwork  may  be  examined. 
Improper  procedures  may  be  thoroughly  evaluated.  Improper  mal- 
function determinations  may  be  traced  back  to  the  point  of  the 
original  mistake.  Technical  publications  and  training  can  be 
methodically  evaluated.  The  adequacy  and  proper  use  of  tools 
may  be  verified. 

Depending  on  how  they  are  used,  video  tapes  may  account  for 
less  test  interference  than  direct  test  observation  alone. 

This  would  be  true  for  an  equal  amount  of  test  data  gathered 
as  a result  of  a relatively  complex  test.  Once  recorded,  the 
data  record  is  permanent  and  may  be  presented  for  use  to  numerous 
persons  including  contractor  and  customer  alike.  The  tapes  may 
be  easily  stopped,  started  and  backtracked  for  repeated  observa- 
tion. Each  task  may  be  thoroughly  examined  step  by  step.  Test 
sequences  that  may  not  be  properly  recorded  may  be  easily  reviewed 
and  retaken. 
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Further  advantages  include  the  fact  that  observer  errors 
are  reduced,  the  observation  can  be  recorded  and  observed 
remotely  from  what  might  be  a hazardous  or  congested  area. 

The  taoes  may  have  considerable  use  as  training  aids.  They 
require  no  time  to  process,  but  motion  picture  films  do. 

The  tape  itself  is  reclaimable;  it  may  be  used  over  and  over 
again  for  different  tests.  The  record  of  time  tags  along  with 
the  video  is  possible. 

Disadvantages  of  the  technique  are  in  the  requirement  for 
special  personnel  or  training  required  to  use  the  recording 
equipment.  The  initial  cost  of  the  equipment  is  quite  high 
(several  thousand  dollars  for  the  recorder,  camera,  zoom  lens, 
monitor,  tripod  and  lights).  Slow  motion  and  stop  action 
shots  are  possible  but  much  more  expensive.  If  necessary,  the 
one  alternative  technique  to  use  is  motion  picture  film. 

References:  McDonell  Douglas  Astronautics  has  used  this  technique  for 

several  years  on  aircraft  systems  such  as  the  F-4E  and  F-15. 

The  following  technical  report  thoroughly  describes  the  use 
of  this  technique  on  the  AN/APW  radar,  Martin  Baker  Mark  H7 
rocket  ejection  seat,  and  Rockeye  MK20  bomb:  Crites,  C.  D. , 

Video  Tape  Recordinn  as  a Technique  for  Personnel  Subsystem 
Test  and  Evaluation,  AFHRL-69-18,  Air  Force  System  Command, 
W-PAFB,  September  1969. 

Additional  information  on  this  technique  as  comparea  with  other 
automatic  recording  techniques  may  be  found  in  Appendix  D. 

5.4.4  Photography 

Description:  This  technique  is  perhaps  too  simple  to  be  considered  as  such 

and  should  be  describeH  rather  as  a HFE  test  and  evaluation 
tool.  It  is,  very  simply,  the  process  of  taking  photographs 
of  whatever  tasks,  objects  or  events  that  are  pertinent  to  the 
HFE  effort.  As  in  the  case  of  the  video  records,  actual  proto- 
type hardware  or  mockups  must  be  available  to  justify  the  use  of 
the  tool.  HFE  test  operators  must  be  familiar  with  the  test  to 
know  when  the  critical  tasks  or  events  require  the  visual  record. 
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In  addition  to  the  camera,  a tripod  and  special  lighting  may 
be  required.  Flash  attachments  are  easily  used.  Depending 
on  facility  and  agency  requirements,  a photographic  pass  may 
be  required.  The  location  of  the  test  may  restrict  the  use  of 
cameras.  Polaroid  type  cameras  are  convenient  in  that  they 
provide  an  instant  picture  for  evaluation  as  to  the  need  for 
additional  pictures.  However,  the  quality  of  the  instant  pic- 
ture cameras  tends  to  be  inferior  to  those  which  produce  the 
large  8 x 10  shots.  The  results  of  the  photography  generally 
are  appropriate  for  inclusion  in  test  reports  or  otner  HFE 
test  and  evaluation  reporting  forms. 

Use/Validity:  Naturally,  photography  is  a well  used  HFE  test  and  evaluation 
tool.  It  is  easy  to  use  and  may  be  done  quickly.  The  parti- 
cular advantages  gained  in  using  this  technique  are  simila'-  to 
some  of  those  for  the  video  tapes  and  motion  pictures,  e.g  , 
the  photograph  is  a permanent  record  which  may  be  reviewed,  it 
may  be  used  as  a training  aid,  and  decreases  observer  errors 
about  what  really  happened.  Photographs  are  used  extensively 
in  HFE  testing  for  analysis  of  anthropometric  interface  problems. 

The  obvious  disadvantage  associated  with  the  use  of  this  T&E  tool 
is  in  the  sinale  frame  static  picture  rather  than  the  dynamic  pic- 
ture created  by  motion  oictures  or  video  taoes.  A small  problem  may 
be  created  by  the  logistics  of  obtaining  the  photographic  equip- 
ment and/or  camera  personnel  and  the  permission  to  use  the  equip- 
ment in  the  test  area.  Alternatives  to  photography  are  the  more 
expensive  video  tapes  or  motion  pictures  or  possibly  a good  fast 
sketcher  assigned  the  duties  of  the  HFE  test  observer.  In  a 
few  instances,  a large  number  of  descriptive  words  written  in  the 
test  reports  may  substitute  for  a photograph  o'^  the  situation 
or  equipment  that  they  are  describing,  but  these  descriptions 
are  seldom  completely  satisfactory. 
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tion  of  the  482L  Emergency  Mission  Support  System.  PSTE, 
APGC-TR-65-61 , AD  476  172,  Volume  III,  Deputy  for  Test  and 
Operations,  Air  Proving  Ground  Center,  Eglin  AFB,  December 
1965. 

b)  Alter,  F.  H. , Ground  Electronic  System  for  WS-1338  (Minuteman), 
PSTE  Plan,  MOP  PD  0060,  Sylvania  Electronic  Systems,  Minuteman 
Program  Office,  Waltham,  Mass.,  March  1964. 

c)  Crites,  C.  D. , Press  Camera  with  Polaroid  Back  Technique 
for  Personnel  Subsystem  Test  and  Evaluation,  AFHRL-69-17, 

Air  Force  System  Command,  W-PAFB,  September  1969. 

d)  Majesty,  M.  S. , Human  Factors  Concepts  for  Testing  Complex 
Man/Machine  Systems,  Headquarters  Ballistics  Systems  Division, 
AFSC,  June  1961. 

Limited  additional  information  on  this  tool  may  be  found  in  Appen- 
dix D. 

5.4.5  Event  Recording 

Description : This  is  a technique  or  method  for  recording  test  situation  or 

event  times.  . The  equipment  involved  in  the  use  of  this  technique 
varies  in  complexity  from  the  stopwatch  to  complete  systems. 

The  more  complex  event  recorder  systems  might  include:  an  event 

recorder,  battery  pack,  event  control  box  and  a signal  cable. 

The  event  recorder  itself  should  be  capable  of  recording  on 
several  channels;  the  battery  pack  is  to  give  portability  to 
the  operation;  the  control  box  is  to  actuate  the  various  channels 
in  the  recorder,  and  the  signal  cable  is  to  electrically  tie  the 
control  box  to  the  recorder.  Other  recording  systems  are  provided 
which  combine  these  units  into  one  easily  portable  package. 

The  sequence  of  events  which  might  occur  with  the  use  of  this 
technique  may  be  as  follows:  HFE  personnel  who  are  to  observe 

the  particular  test  first  become  familiar  with  the  planned  test 
events.  They  estimate  what  tasks  are  more  critical  and  should  be 
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recorded  In  terms  of  time  performance.  If  the  tasks  to  be 
monitored  are  particularly  critical  they  may  even  perform  a 
dry  run  of  the  test  or  plan  to  run  multiple  replications  of  the 
time  critical  task.  The  total  test  may  be  divided  into  several 
functional  tasks  and  each  such  assignment  allocated  to  a separate 
channel.  Examples  of  such  task  functions  are  reading  technical 
publications,  actuating  controls,  reading  displays  and  making 
adjustments.  The  channel  controls  are  easily  activated  for  each 
j of  the  task  functions  as  they  start  and  stop.  It  may  be  necessary 

to  write  start  labels  for  each  event  on  each  of  the  channels 
plotted  on  the  recorder  chart  paper  roll . Figure  5.4-1  shows  a 
I sample  of  this  type  of  annotated  record  (See  Reference  (a)  this 

f 

! section). 

I 
I 

More  recently  available  recording  equipment  does  not  require  the 
use  of  the  paper  role  for  a record  of  events.  The  test  observer 
simply  press  combinations  of  keys  to  note  task  functions  as  they 
occur.  Data  entries  record  in  a solid-state  memory  in  a computer 
program  format.  The  data  is  later  transmitted  to  the  computer  by 
connecting  the  device  via  a simple  connecting  cable.  In  this 
manner,  computer  written  reports  may  be  written  in  minutes  (See 
Reference  b).  This  device  includes  a space  for  written  notes 
on  an  integral  note  pad. 

Another  system  requires  the  use  of  multiple  keyboards  with  ten 
keys  (or  pushbuttons)  on  each  keyboard.  This  system  interfaces 
with  a computer  through  the  use  of  a magnetic  tape  recorder 
(See  Reference  c). 

The  direct  outputs  of  each  of  these  event  recording  techniques 
varies  from  handwritten  notes  to  complete  computer  printouts 
of  evaluated  data.  The  eventual  outputs  are  verification  of 
task  time  data. 
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Start  of  event 
on  channel  #3 


■OTATION  13-7 

13  seconds  7^  occurrenpa 


W>TATI^|I  7-8 

7 seconds  8—*  occurrence 


Figun  5.4-1:  Sample  Annotated  Event  Record 


I 


I 

5 

I 
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Use/Validity;  Most  HFE  test  and  evaluation  efforts  will  require  the  use  of 


one  of  the  following  (but  previously  indicated)  event  record- 
inn  techniques  or  some  variation  thereof: 

a)  Event  recorder  and  separate  control  box  (Reference  a). 

b)  Combined  function  solid  state  memory  data  collector 
(DATAMYTE,  Reference  b). 

c)  Multiple  keyboard  entry  to  mag  tape  records  DACOLS, 

Reference  c). 

d)  Stopwatch. 

When  critical  test  events  must  be  recorded  and  evaluated,  these 
techniques  prove  valuable  for  determining  system/operator  time 
performance  capabilities.  With  the  exception  of  a single  stop- 
watch these  techniques  allow  several  task  functions  to  be 
recorded  at  once.  The  observer  may  thereby  direct  more  of  his 
attention  to  the  other  aspects  of  the  test.  The  stopwatch  is, 
of  course,  by  far  the  cheapest  method  of  the  four  of  recording 
time.  It  may,  upon  occasion,  be  the  most  cost  effective.  It  is, 
however,  more  error  prone  than  the  other  methods.  The  recordings 
made  from  the  later  three  techniques  (a,  b,  and  c)  can  be  used 
for  timeline,  task  loading  and  time  sharing  analysis. 

The  disadvantages  of  the  first  three  techniques,  when  comoared 
to  the  stopwatch,  are:  the  cost,  requirement  for  a test  with 

several  different  task  function  channels  occurring  simultaneously 
to  be  useful,  and  ease  of  use.  Technique  "b"  is  better  than  "a" 
and  "c"  in  that  it  is  easily  portable,  immediately  compatible 
with  existing  computer  programs,  and  includes  an  earphone  timer 
tone.  Technique  "c"  is  better  than  "a"  in  that,  like  "b",  it 
allows  the  rapid  conversion  of  data  to  computer  programs. 

In  general,  all  techniques  will  measure  objectively  human  per- 
formance and  provide  useful  data  for  the  test  as  a whole.  The 
techniques  can  be  used  with  very  little  test  interference.  The 
training  required  to  use  the  technique  equipment  varies  with  the 
equioment  complexity  but  is  generally  uncomplicated.  The  data 
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is  applicable  for  time  to  accomplish  tasks,  evaluation  and 
optimization  of  tasks  involving  team  work,  and  the  isolation 
of  specific  points  that  degrade  turn-around  times,  loading 
times  and  launch  times.  The  technique  may  not  be  used  for 
evaluation  per  se,  but  further  analysis  must  be  made  of  the 
data  using  other  techniques. 


References : a)  Crites,  C.  D. , Miniature  Event  Recording  as  a Technique 

for  Personnel  Subsystem  Test  and  Evaluation,  AFHRL-69-16, 
Air  Force  System  Command,  W-PAFB,  September  1969. 

b)  Electro/General  Corporation  Datamyte  Data  Collector, 
Minnetonka,  Minnesota  55343. 

c)  Potempa,  K.  W. , "Data  Collection  Systems  (DACOLS)", 

A Catalog  of  Human  Factors  Techniques  for  Testing  New 
Systems , AFHRL-TR-68-5,  AD  8544821,  February  1969. 


Additional  comparison  information  on  this  technique  as  compared 
to  other  automatic  recording  techniques  may  be  found  in  Appendix  D. 


5.4.6  Operational  Performance  Recording  and  Evaluation  Data  System  (OPREDS) 

Description:  OPREDS  is  a real  time  data  acquisition  system  which  presently 

is  being  used  to  evalucte  the  Naval  Tactical  Data  System  (NTDS) 
and  related  command  and  control  system  performance.  OPREDS 
consists  of  two  major  system  components:  a computer  controlled 

data  acquisition  system  which  collects  NTDS  and  non-NTDS  data; 
and  a computerized  data  management,  analysis  and  information 
system.  OPREDS  performs  analyses  and  evaluations  of  ship 
systems  in  two  phases: 

a)  Onboard  data  recording  and  preliminary  analysis  using 
the  Data  Collection  System  (DCS). 

b)  Data  reduction,  analysis,  storage  and  retrieval  using 
the  OPREDS  Data  Reduction  System  on  a shore-based  IBM 
System/360  computer. 

Together,  these  subystems  permit  the  reconstruction  and  per- 
formance evaluation  of  a complete  exercise  of  any  duration. 
Additionally,  the  onboard  analysis  feature  permits  an  evaluation 
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team  to  conduct  preliminary  assessments  of  performance  in  j 
order  to  design  subsequent  exercises  and  conduct  operational  i 
tests.  I 


The  Data  Collection  System,  the  shipboard  component  of  OPREDS, 
consists  of  a small,  specialized  minicomputer  and  tape  transport. 
These  components  can  acquire  and  filter  all  data  flowing  between 
the  NTDS  unit  computer  and  the  display  consoles.  Additionally, 
data  sources  external  to  the  NTDS  system  can  be  collected  and 
processed  using  Remote  Input  Devices  (RID).  The  entire  data 
acquisition  and  recording  process  is  done  in  real  time  and  in 
no  way  impacts  or  degrades  the  system  under  study  nor  the 
performance  of  personnel  associated  with  its  operation.  The 
OPREDS  DCS  has  been  carefully  designed  for  this  environment  and 
includes  the  following  characteristics: 

a)  OPREDS  does  not  have  to  be  in  immediate  proximity  to  the 
combat  team;  thus,  the  data  collection  process  does  not 
interfere  with  their  functions.  In  many  cases,  they  are 
not  aware  that  operations  are  being  monitored. 

b)  OPREDS  does  not  require  any  assistance  from  the  combat 
system's  equipment. 

c)  OPREDS  is  small,  transportable, and  can  be  located  in  any 
convenient  location. 

d)  OPREDS'  DCS  operating  programs  are  firmware.  They  are  in- 
destructible, impervious  to  interruptions  and  require 
virtually  no  operator  intervention. 

The  OPREDS  computer  can  also  be  utilized  as  a stand-alone, 
general  purpose  minicomputer  when  not  in  the  data  collection 
mode. 

The  OPREDS  Data  Reduction  System  (DRS)  component  consists  of  a 
large  software  system  operational  on  IBM  System/360  computers. 

The  DRS  decodes  the  raw  data  tapes  recorded  by  the  onboard 
system,  reconstructs  the  sequence  of  events  as  they  occurred 
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during  the  exercise,  develops  track  plots  and  histories,  and 
action  listings  with  English  language  descriptors  of  each 
operator,  console  action  and  mode,  as  well  as  a variety  of 
statistical  analyses. 

OPREDS  has  an  alternative  configuration  allowing  for  limited 
onboard,  post-exercise  analysis.  Using  the  OPREDS  DCS,  the 
system  can  be  configured  to  yield  track  summaries,  relative 
ranges  and  bearings  between  threats  and  weapons  and  other 
key  measures. 

Because  of  the  unique  capability  of  the  DCS  to  capture  all 
system  data,  and  the  generalized  management  information  system 
construction  of  the  DRS,  evaluation  and  effectiveness  measures 
can  be  derived  from  the  data  with  relative  ease. 

The  following  data  are  recorded  by  the  DRS  on  magnetic  tape, 
or  can  be  determined  from  displays  on  the  control  panel: 

a)  Identification  of  each  console. 

b)  Mode  of  each  console. 

c)  Identification  of  each  quick  action  button. 

d)  Identification  of  each  function  code. 

e)  Content  of  each  function  code. 

f)  Time  of  each  action. 

g)  Link  eleven  track  numbers. 

h)  Own  ship  track  numbers. 

i)  Console  on  which  each  track  displayed. 

j)  Number  of  illegal  actions  per  console. 

k)  Identifier  information  (Date,  time,  ship, 
observer,  etc.). 


In  addition,  situational  variables  must  be  identified  and 
carefully  rated  by  the  observer  for  interpretation  and 
multiple  correlation  analyses  following  the  recording.  To 
permit  these  correlations,  each  variable  will  carry  a code 
as  a data  ooint  for  future  sorting: 


GENERAL: 

Ship  Class 
Sea  States 
Threat  immediacy 
Time  on  station 
Time  on  watch 
Time  of  day 
OPERATOR  FACTORS: 
Training 
Schools 
On-the-job 
Westpac  experience 
Time  in  service 
Rating 

Previous  sleep 
Previous  duty 


TEAM  FACTORS: 

Time  together 
Full  team 
Degraded  team 
New  members 
SENSOR  FACTORS: 

Degraded  radar 
Degraded  comm. 

NTDS  FACTORS: 

Dropped  program 
Restart  program 
Console  modes  out 
JUDGMENTS: 

Operator  opinions 
Supervisor  opinions 
(Was  it  tough  or  easy?) 


Results  of  the  data  reduction  process  are  summaries  of  the 
number  of  actions  and  their  times  for  each  operator  and  the 
CIC  team.  Examples  are:  number  of  tracks  handled,  separate 

actions,  and  submode  changes.  By  subtraction  and  bracketing, 
times  can  be  known  for  events,  such  as  seconds  between  each 
action  at  each  console,  time  spent  on  each  track  number  processed, 
and  time  to  firm  track,  identify  track,  assign  weapons  and  eval- 
uate threats. 


A second  output  from  this  module  are  various  types  of  listings 
of  sequences,  such  as  all  of  the  actions  and  errors  that  occur 
in  turn  for  each  track  number  or  combination  of  track  numbers, 
or  the  series  of  actions  that  occur  at  each  console. 
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The  second  program  module  analyzes  the  data  for  statistical 
information,  averages,  variability,  relationships,  and  searches 
for  any  series  of  actions  that  deviate  from  doctrine  or  are  in- 
efficient for  the  utilization  of  the  NTDS  system.  Outputs  that 
result  from  the  analyses  are  curves,  histograms,  averages, 
correlations,  exercise  reconstructions  by  continuous  listings, 
by  track  number,  or  sets  of  track  numbers  and  reconstructions 
in  OSD  format. 

Use/Val idity:  Prior  to  development  of  this  system,  there  was  no  practical 

method  for  use  in  the  operational  environment  to  measure  response 
time  and  to  record  each  of  the  operator  actions  from  each  NTDS 
console  with  the  related  track  numbers,  and  the  number  of 
illegal  action  errors  that  are  made. 

Data  obtained  from  the  DRS  in  a shipboard  environment  could  be 
used  to  establish  the  proficiency  of  NTDS  team  members.  With 
this  information  available,  a team  of  highly  proficient  operators 
could  be  assigned  for  use  during  peak  load  situations  (multi- 
track/high density)  and/or  hostile  engagement  phases.  With 
each  operator  rated  according  to  established  performance 
standards,  on-the-job  training  requirements  for  individual 
operators  would  be  clearly  defined.  Concerted  training  could 
be  directed  to  those  operators  that  are  below  the  performance 
standards. 

Until  the  development  of  the  DRS,  effectiveness  of  training  pro- 
vided to  NTDS  operator  personnel  was  based  primarily  on  judg- 
ments made  by  instructors,  and  to  a limited  degree  on  quanti- 
tative data  extracted  from  various  computerized  data  reduction 
programs.  The  installation  of  the  DRS  at  NTDS  training  activi- 
ties, such  as  the  Fleet  Anti-Air  Warfare  Training  Centers,  could 
be  used  to  measure  NTDS  console  operator  performance  during  the 
period  of  time  that  the  student  is  being  provided  instruction. 
Using  the  DRS,  operator  performance  can  be  measured  and  a compar- 
ison made  between  fleet  performance  standards  and  training  per- 
formance standards  achieved  during  the  training  period. 
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From  the  above  analyses  and  outputs,  evaluation  will  be  made 
as  to  the  level  and  efficiency  of  performance.  The  source  of 
major  delays  can  be  examined  as  to  who  caused  them,  what  they 
were,  who  was  involved,  how  they  differ  from  week-to-week, 
watch-to-watch  and  supervisor-to-supervisor. 

When  used  with  UYK-7  on  shipboard,  immediate  evaluations  of 
team  performance  is  possible.  By  rating  operators  to  pre- 
established  standards,  and  from  day-to-day  monitoring,  their 
operational  readiness  can  be  evaluated  for  the  establishment 
of  a highly  proficient  "Critical  Situation"  team. 

In  the  last  two  years,  OPREDS  has  demonstrated  its  effectiveness 
in  evaluating  NTDS  performance  in  a variety  of  shipboard  environ- 
ments. These  ships  include  the  USS  Independence  (CV-62),  USS 
England  (DLG-21),  USS  Towers  (DDG-9,  JPTDS  ship),  the  Fleet 
Anti-Air  Warfare  Training  Center  (FAAWTCPAC),  the  NELC  ASOEC 
Test  Site  and  several  other  DLGs  conducting  training  exercises 
with  the  dockside  T1  van.  OPREDS  can  interface  with  all  current 
NTDS  systems  and  consoles.  It  has  been  used  with  SYA-1 , SYA-4, 
UYA-4,  and  OJ-194  consoles  on  both  NTDS  and  JPTDS  (JEEPS) 
equipped  ships. 

The  Operational  Performance  Recording  and  Evaluation  Data  System 
(OPREDS) , Barten  and  Associates,  Santa  Monica,  California, 

June  1975. 

The  DRS  was  developed  by  Jakus  Associates  (E.  B.  Morrison  and 
J.  R.  Harris)  and  the  OPREDS  was  designed  by  Barten  & Associates 
(B&A)  under  contract  to  the  Naval  Electronics  Laboratory  Center 
(NELC),  San  Diego,  California.  Additional  information  on  this 
technique  as  compared  to  other  automatic  recording  techniques 
may  be  found  in  Appendix  D. 
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5.4.7  Secondary  Task  Monitorin' 


Description:  For  the  purpose  of  determining  crew  workload,  test  parti ci-  | 

pants  are  given  both  operational  tasks  and  secondary  tasks 

to  perform.  The  secondary  tasks  may  or  may  not  be  meaning-  | 

less  in  relation  to  the  rest  of  the  test  set  up.  They  are, 

however,  in  no  way  necessary  to  the  operational  tasks  being  ■ 

tested.  The  secondary  tasks  are  performed  with  prototype 
hardware  or  hot  mockups  on  special  equipment  that  is  instru- 
mented through  hardwire  or  telemetry  to  record  crew  perform-  i 

ance.  The  participant  is  instructed  to  perform  the  secondary 
tasks  when  not  required  to  perform  the  operational  tasks. 

The  time  taken  to  perform  the  secondary  tasks  is  recorded  and 
subtracted  from  the  total  time  available.  In  this  manner,  the 
crew  workload  required  to  perform  the  operational  tasks  is 
implied  on  the  basis  of  the  measured  time  (or  effort)  not  spent 
doing  those  same  operational  tasks. 

iJse/Val idity : This  is  a useful  technique  to  measure  crew  workload  particu- 

larly when  it  is  not  feasible  to  monitor  directly  the  opera- 
tional performance  parameters.  Because  workload  can  be  quanti- 
tatively measured  in  this  case,  it  can  be  more  accurate  than 
many  other  workload  evaluation  techniques.  The  cost  and  effort 
to  implement  this  technique  is  relatively  high  as  compared  to 
several  other  HFE  T&E  techniques  if  the  secondary  task  data  is 
recorded  automatically.  However,  the  cost  is  inherently  lower 
than  monitoring  operator  performance  on  each  of  the  operational 
controls  (and,  if  possible,  displays). 

There  are  two  basically  different  types  of  secondary  task  monitor- 
ing. The  first  type  uses  secondary  tasks  that  are  completely  un- 
related to  the  system  operational  tasks.  These  are  make-work 
tasks.  The  second  type  is  more  sophisticated  in  that  the  secon- 
dary tasks  are  essentially  the  same  as  the  required  opera- 
tional tasks.  Test  participants  seem  to  have  more  motivation  to 
do  the  more  real  secondary  tasks  rather  than  the  make-work  tasks. 
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Additional  data  on  this  technique  as  compared  with  other 
techniques  may  be  found  in  Appendix  D. 

5.5  Physiological 

5.5.1  Physiological  Instrumentation 

Description:  The  process  of  measuring  test  participant  physiological  data 

is  generally  guite  rigorous.  In  addition  to  all  of  the  set 
up  procedures  required  for  the  test  itself,  it  requires 
several  important  tasks  that  must  be  performed  just  for  the 
physiological  instrumentation. 

Physiological  measurement  requires  more  commitment  from  the 
test  participants.  The  purpose  of  the  instrumentation  may  be 
to  monitor  physiological  parameters  to  insure  that  the  parti- 
cipant remains  in  a safe  range  of  performance.  The  implication 
of  this  is  that  there  is  a possible  unsafe  range  of  performance 
and  therefore  more  commitment  required  on  the  part  of  the  test 
participant.  Even  if  this  is  not  the  case,  the  encumbrances  of 
the  test  sensors  on  the  participant  are  generally  somewhat 
annoying. 

Trained  medical  personnel  must  approve  the  test.  Generally, 
they  should  perform  the  test  set  up  of  the  instrumentation  system. 
This  would  involve  the  attachment  of  the  sensors  to  the  partici- 
pants. Care  must  be  taken  to  place  the  sensors  in  a manner  to 
minimize  their  effect  on  the  total  test.  Medical  personnel  must 
also  be  present  during  the  test  if  any  participant  risk  is 
invol ved. 


nn 


Electronics  technicians  may  also  be  required  to  adjust  the 
test  instruments. 


Use/Validity: 


In  addition  to  the  individual  parameter  sensors  located  on 
the  participant,  wire  leads  must  be  provided.  Attached  to 
the  leads  would  be  the  appropriate  transmitters  (if  telemetered), 
receivers  and/or  amplifiers.  Instruments  for  displaying 
each  of  parameter  values  and  chart  recorders  will  also  be 
required. 

Parameters  that  might  be  monitored  are  as  follows: 

a)  heart  rate,  blood  pressure 

b)  respiration  rate,  volume 

c)  galvanic  skin  response  (GSR) 

d)  electroencephalograph  (EEG) 

e)  electrocardiograph  (EKG) 

f)  body  temperature 

g)  body  movement. 

Upon  completion  of  the  test,  medical  personnel  are  required 
for  analysis  and  evaluation  of  the  resulting  test  physio- 
logical data. 

Physiological  measurement  is  performed  much  more  for  research 
testing  than  for  operational  or  field  type  testing.  It  is 
also  used  when  there  is  a possibility  of  risk  involved,  for 
example,  centrifuge  runs.  Physiological  testing  is  seldom 
intended  to  measure  total  system  performance,  let  alone  the 
more  normally  monitored  operator  performance  parameters  of 
time  and  errors. 
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The  cost  to  oerform  this  type  of  testing  is  relatively  high 
and  the  effort  involved  by  HFE,  medical  and  technical  per- 
sonnel is  considerable.  Because  of  the  nature  of  the  test 
itself,  which  would  reouire  the  use  of  physiologi cal  instru- 
mentation for  safety,  the  testing  must  be  considered  to  be 
performed  on  an  interference  basis.  When  ohysiological  mon- 
itoring is  really  needed,  there  is  no  substitute  technique 
that  may  be  used  to  obtain  the  necessary  data.  The  only 
alternative  of  constantly  stooping  the  test  to  take  time  out 
for  the  required  measurements  is  unacceptable.  By  use  of 
radio  transmitters,  the  technique  nay  be  monitored  remotely 
away  from  the  test  area.  The  most  notable  use  of  this  tech- 
nique has  been  in  manned  space  programs,  i.e.,  Skylab,  Apollo, 
Gemini  and  Mercury. 

References : a)  Zonjer,  F.  H. , "The  Contribution  of  Work  Physiology  to  the 

Evaluction  of  Man-Machine  Systems,"  Measurement  of  Man  at 
Work,  Taylor  and  Francis,  London,  1971. 

b)  Brown,  C.  C, , and  Saucer,  R.  T.,  Electronic  Instrumentation 
for  the  Behavioral  Sciences,  C.  C.  Thomas,  Springfield, 
Illinois,  1958. 

c)  Glassner,  H.  F. , and  Peters,  G.  A.,  "Bio-Electronic  Analysis 
of  Performance",  PAC  Engineering  Paper  No.  897,  AD  235  969, 
Equipment  and  Sa-^ety  Research  Section,  Douglas  Aircra-'^t 

Co . , Inc . , March  1 960. 

d)  Jacobs,  E.  p.  , Second  Interim  Report,  Test  Instrumentation 
for  Pilots  in  High  Performance  Aircraft  to  Evaluate  Cockpit 
Environments  and  Personnel  Physiological  Equipment,  Report 
No.  ST35-43R-63,  AD  480  526,  Naval  Air  Test  Center,  Sep- 
tember 1963.  Lindsey,  J.  F.,  et  al . , Proposed  Pilot  Model 
for  Combat  Effectiveness  Testing,  PGN  Document  62-7,  Deputy 
for  Bioastronautics,  Air  Proving  Ground  Center,  Eglin  AFB, 
August  1962. 

Additional  information  on  this  technique  as  compared  with  others 
m.ay  be  found  in  Appendix  D. 
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5.5.2  Physical  Measurement 

Description:  This  technique  is  the  process  of  measuring  what  the  test  parti- 

cipants can  do  in  terms  of  their  physical  performance  or  what 
they  are  doing  in  terms  of  physical  and  cognitive  performance. 

Three  different  types  of  physical  measurement  are  presented 
in  this  section.  The  first,  anthropometry,  deals  with  potential 
test  participant  physical  performance.  The  other  two,  oculometry 
and  voice  monitoring,  pertain  to  measurement  of  the  participants’ 
physical  and  cognitive  processes. 

Anthropometry.  Anthropometric  measurements  may  be  made  of  each  of 
the  test  subjects  to  be  used  in  a hardware  prototype  or  mockup 
test.  These  measurements  are  taken  on  the  assumption  that  the 
test  will  indicate  various  areas  of  work  space  or  work  access  veri- 
fication. If  problems  are  indicated,  rather  than  designs  veri- 
fied, then  detailed  measurements  are  taken  as  to  exactly  how  much 
of  a work  space  problem  exists.  If  much  of  the  test  is  to  hinge 
on  the  ability  of  the  test  participants  to  fit  the  equipment  (e.g., 
cockpit  egress),  the  subjects  may  be  specially  screened  and  chosen 
to  fit  the  worst  case  (larger)  population  percentiles  (95th  or  98th 
percentile).  If  a subject  with  98th  percentile  buttock-knee  length 
and  98th  percentile  shoulder  breadth  can  successfully  egress  with 
the  given  cockpit  dimensions,  then  it  may  be  assumed  that  most 
pilots  will  be  able  to  do  the  same  at  least  in  terms  of  egress 
space. 

Oculometry.  This  is  the  tecnnique  of  measuring  the  test  partici- 
pant's eye  movement  while  he  is  seated  at  (in)  a mockup  or  proto- 
type hardware  of  the  system  being  tested.  The  oculometer  is  used 
to  view  the  participant's  eye  movement  in  terms  of  deflection  rate 
and  amount.  The  instrument  and  associated  equipment  is  capable 
of  recording  the  links  between  controls  and  dipslays,  the  dwell 
times  on  each,  the  total  number  of  eye  contacts,  and  the  probabil- 
ity of  next  contact.  The  oculometer  performance  is  at  a half 
degree  at  30  inches  from  the  eye  within  an  envelope  30°  up, 
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10°  down,  and  60°  horizontal.  Once  this  data  is  recorded,  panel 
layout  adequacy  is  verified  by  the  quantity,  location  and  rate 
of  eye  movements. 


Voice  Monitoring.  This  technique  is  performed  as  a means  of  psy- 
chological stress  evaluation.  By  the  use  of  sophisticated  voice 
monitoring  equipment,  similar  to  that  being  used  for  lie  detection, 
the  voice  is  analyzed  to  determine  stress.  The  stress  indicates 
test  situations  where  the  participant  is  having  problems  or  is 
close  to  the  point  of  having  problems.  The  voice  stress  analysis 
equipment  requires  operation  by  trained  evaluators.  These  evalua- 
tors should  be  familiar  with  the  system  test  objectives  in  order 
to  be  better  able  to  analyze  test  data  and  to  recommend  problem 
solutions. 

Physical  measurements  may  also  include  participant  muscular  strength, 
body  weight,  limb  coordination,  visual  and  auditory  acuity,  and 
kinesthetic  response. 

Use/Validi ty:  Anthropometry.  It  is  relatively  easy  to  measure  test  participants 

to  determine  their  anthropometric  measurements.  The  fact  that 
these  subjects  either  did  or  did  not  fit  the  particular  mockup  or 
prototype  is  also  easy  to  note  and  record.  The  difficulty  in  the 
use  of  this  technique  is  if  and  when  particular  anthropometric  di- 
mensions are  required  as  test  subjects.  It  is  very  difficult  for 
HFE  observers  to  go  out  and  find  particular  anthropometric  dimen- 
sional subjects,  particularly  for  combinations  of  measurements  and 
for  the  extremes  of  the  population  (e.g.,  greater  than  90th  per- 
centile and  less  than  10th  percentile). 

The  real  value  in  using  anthropometric  measurements  is  in  the  knowl- 
edge of  how  close  the  design,  as  represented  by  the  mockup  or  proto- 
type, comes  to  the  specified  user  anthropometry.  The  disadvantage 
is  the  effort  in  finding  subjects  who  properly  represent  the  required 
population.  If  this  technique  is  not  used  and  work  space  clearances 
are  critical  to  the  test  conduct,  the  HFE  observer  runs  a high  risk 
in  only  guessing  the  anthropometric  characteristics  of  the  test 
participants. 
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Oculometry.  The  oculometer  technique  is  relatively  complex  and 
expensive  to  use.  It  cannot  be  run  on  a non-interference  basis. 

It  requires  trained  HFE  observers  to  use.  The  use  of  the  tech- 
nique is  still  somewhat  experimental.  The  major  advantage  in 
the  use  of  the  technique  is  that  it  is  the  ideal  way  to  perform 
or  verify  cockpit  or  console  panel  link  analysis  data.  If  not 
used,  questionnaires  or  interviews  may  be  used  to  determine  sub- 
ject reaction  to  panel  layout  adequacy. 

Voice  Monitoring.  The  use  of  voice  monitoring  is  both  experimental 
and  controversial.  Like  the  oculometer,  it  is  a comolex  technique. 
It  requires  trained  evaluators  and  special  equipment  and  is  there- 
fore expensive.  Interpretation  of  the  test  participant  voice  qual- 
ities is  variable.  On  the  plus  side,  the  technique  may  reveal  prob- 
lems that  no  other  technique  could  uncover.  The  only  alternatives 
to  its  use  are  interviews  and  questionnaires  to  try  and  dig  out 
stressful  test  situations.  This  technique  has  been  used  in  pilot 
evaluation  during  aircraft  carrier  night  landings. 

There  are  several  anthropometry  references  including  the  techniques 
of  CAPE,  CAR,  CGE,  Environmental  and  Perforamnee  Measuring  Equip- 
ment, and  G-2/G-5  Anthropometers . Text  book  data  may  also  be  found 
in  numerous  publications  including  Van  Cott,  H.  P.,  and  Kinkade, 

R.  G.,  Human  Engineering  Guide  to  Equipment  Design,  sponsored 
by  Joint  Army-Navy-Air  Force  Steering  Committee,  U.S.  Government 
Printing  OFfice,  1972.  MIL-STD-1472B,  Human  Engineering  Design 
Criteria  for  Military  Systems,  Equipment  and  Facilities,  also 
includes  a large  amount  of  anthropometric  data. 

Additional  oculometry  data  may  be  found  in  NASA  reports  NASA-CR- 
149951  by  Honeywell  (Boston)  and  NASA-TM-X-3344  by  Jim  Wise. 

A recent  Honeywell  report  is  titled  An  Oculometer  Now  Unobtrusively 
Reveals  What  You  Are  Looking  At. 

The  subject  of  speech  intonations  is  covered  in  a report  by  Luk- 
'yanov,  A.  N. , and  Frolov,  M.  V.,  Signals  of  Human  Operator  State, 
NASA-TT-F-609,  1969. 


115 


5.6  Simulation 


5,6.1  Cockpit  Geometry  Evaluation 


Description:  The  Cockpit  Geometry  Evaluation  (CGE)  Program  is  an  experi- 

mental development  partially  funded  by  the  Joint  Army-  Navy  Air- 
craft Instrumentation  Research  (JANAIR)  Program  Working  Group  to 
develop  an  improved  method  of  evaluating  the  physical  compati- 
bility of  crewmen  with  specified  crew  stations.  The  heart  of 
the  program  is  a computerized  mathematical  man-model  (BOEMAN). 

The  CGE  Computer  Program  System  (CGECPS)  serves  as  the  basic 
integrated  tool  to  store  required  anthropological  and  geometric 
data,  make  computations  to  simulate  typical  crew  movements  during 
the  performance  of  tasks,  check  and  correct  for  visual  and  physi- 
cal interference,  and  output  related  information.  Crew  station 
compliance  with  selected  military  standard  and  specification 
is  also  performed. 


Using  the  baseline  Computer  Program  System  developed  in  the  first 
phase  of  the  program,  the  second  phase  of  this  project  added  a 
three-dimensional  "human"  form  to  the  first  phase  stick  man. 

Also  provided  were  leg  movement  capability  and  a crew  station 
component  description  sufficient  to  enable  a generalized  physical 
interference  detection  procedure  to  function  effectively.  The  23 
pin- joint  link-man  received  three-dimensional  body  segments 
surrounding  his  links.  The  length  and  mass  of  the  segments  re- 
mained variable,  so  that  a crewman  with  any  link  length  can  be 
approximated.  Currently,  the  breadth  and  width  of  the  body 
segments  remain  fixed  at  50«  values  since  they  influence  crew 
operability  significantly  less  than  link  lengths  and  such  a tech- 
nique helps  standardize  evaluation  results.  A mathematical  op- 
timization routine  is  used  to  simulate  movement  by  the  man-model 
in  a given  crew  station  configuration. 


A later  phase  of  this  project  provided:  increased  efficiency  in 

generating  man-model  positions,  variable  joint  angles  to  more 
closely  resemble  human  limits  of  bend  on  the  links,  physical 
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restraint  simulation  of  lap  belts  and  shoulder  harnesses,  a 
preliminary  physical  interference  avoidance  procedure  to 
eliminate  hand  and  foot  interferences  during  the  task,  examin- 
ation of  compliance  checks  with  MIL-STDs  and  SPECs,  and  evalua- 
tion of  an  existing  aircraft  cockpit  for  physical  compatibility 
problems. 

The  latest  phase  of  this  project  provides:  expansion  of  the 

MIL-STD  and  MIL-SPEC  requirements  compliance  checking  program, 
analysis  of  additional  applications,  improvement  and  scope 
reductions  of  the  Computer  Program  System,  continued  and  ex- 
panded crew  station  evaluation,  continued  man-model  improvements 
in  accuracy  and  speed,  and  continued  refinements  in  storage, 
timing  and  the  capabilities  of  the  CGECPS. 

Use/Val idi ty:  The  CGECPS  may  be  used  to  evaluate  a crew  station  with  a given 

sized  man-model  (BOEMAN)  performing  a task  sequence.  Each  task 
may  be  pre-checked  for  reach  envelope  compatibility.  Straight 
line  hand  and  foot  trajectories  may  be  initially  prescribed  to 
perform  the  task.  If  there  are  interferences  with  any  trajectory, 
a new  segmented  linear  trajectory  can  be  calculated  to  try  to 
avoid  the  interference.  Joint  and  body  segment  vertex  locations 
and  link  and  body  segment  orientations  are  synthesized  at  various 
positions  of  each  task.  Then  the  resulting  positions  are  tested 
for  the  occurrence  of  physical  interference  and  the  final  position 
of  each  task  is  additionally  checked  for  visual  interference. 
Finally,  numerical  performance  data  used  for  further  comparison 
purposes  for  different  crew  station  designs  or  different  sized 
individuals  or  similar  tasks  are  calculated. 

The  capability  of  storing  and  retrieving  anthropological,  geometric 
and  flight  mission  data  is  built  into  the  system.  Successful 
running  of  the  Cockpit  Geometry  Evaluation  Computer  Program  System 
has  been  accomplished,  validations  have  been  performed  against 
human  movement  criteria,  and  the  evaluation  of  the  A-7E  aircraft 
for  52  different  tasks  has  been  made  for  3rd,  5th,  95th  and  98th 
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percentile  BOEMAN.  The  A-7E  aircraft  shows  that  significant 
physical  compatibility  problems  that  exist  in  a real  aircraft 
can  be  discovered  by  this  evaluation  method  and  that  they 
should  be  able  to  be  detected  beginning  with  the  conceptual 
phases  of  design. 

Further  improvements  to  CGE  are  required.  Improvements  to  the 
computer  time  required  to  synthesize  task  performance  and 
scrutinize  a greater  number  of  tasks  are  still  necessary,  if 
the  tool  is  to  be  used  with  maximum  utility.  Actual  and  total 
evaluation  of  proposed  crew  stations  with  unknown  physical 
compatibility  problems  may  next  ^e  made  to  validate  and  improve 
the  elements  of  the  system,  as  well  as  begin  to  provide  the  best 
of  several  designs  and/or  indicate  problems  with  a given  design 
much  sooner. 

CGE  is  more  of  a design  technique  than  a test  and  evaluation 
technique.  It  may  be  used  at  any  time  after  the  DSARC  I de- 
cision date  but  it  is  best  used  prior  to  formal  test  and  eval- 
uation. The  advantages  to  using  CGE  may  be  summarized  as  fo.iows: 
the  technique  may  be  used  for  dynamic  problems,  a large  dat  lank 
is  available  to  draw  on,  the  program  will  optimize  the  problem 
solution,  any  percentiles  of  the  given  population  may  be  tested, 
and  the  anthropometric  model  is  extremely  sophisticated. 

The  disadvantages  associated  with  the  use  of  CGE  are  that:  it 
operates  in  batch  mode  only;  compared  with  other  computer  sim- 
ulations, it  is  complex;  it  requires  special  training  to 

•i 

understand  and  use;  and  it  is  costly  - especially  when  compared  f 

I 

to  an  experienced  HF  engineer's  judgment  of  similar  situations.  j 

[ 

a)  Katz,  R.,  Cockpit  Geometry  Evaluation,  Phase  II  Final  i 

Report,  Volume  III:  Computer  Program  System,  D1 62- 101 2703,  | 

The  Boeing  Co.,  JANAIR  Report  720402,  November  1971.  | 

b)  Whitmore,  D.  C.,  and  Parks,  D.  L.,  Computer  Aided  Function 
Allocation  Evaluation  System,  Phase  IV,  Vol . 1,  0180-18433-1, 

Boeing  Aerospace  Company,  Oecember  1974. 
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Additional  data  on  anthropometric  techniques,  including  CAR 
and  CAPE,  may  be  found  in  other  sections  of  this  guide. 

Appendix  D contains  evaluation  sheets  on  this  technique  as 
well  as  others. 

5.6.2  Crewstation  Assessment  of  Reach  (CAR) 

Description:  The  CAR  Model  establishes  the  actual  percentage  of  naval  I 

aviators  that  can  be  accommodated  in  the  critical  areas  of  a 
given  crewstation  geometric  configuration.  The  model  examines 
hand  and  leg  control  positions,  seat  movement  to  establish 
over-the-nose  vision,  and  head  clearance  for  a representative 


sample  of  the  naval  aviator  population  and  determines  the 
fol lowing: 

a)  The  percentage  of  naval  aviators  that  are  able  to 
position  themselves  at  the  horizontal  olane  of  the 
Design  Eye  Point  (DEP)  within  given  limits  of  hori- 
zontal and  vertical  seat  adjustment. 

b)  The  percentage  of  aviators  that  have  a specified 
minimum  "above  the  head"  clearance  while  positioned 
as  close  to  the  plane  of  the  DEP  as  possible. 

c)  The  percentage  of  aviators  that  are  capable  of  reach- 
ing the  primary  hand  and  leg  controls  in  a crewstation 
while  positioned  as  close  to  the  DEP  as  possible. 

d)  The  amount  of  relocation  required  on  specific  controls 
to  accommodate  a specified  percentage  of  aviators. 

Reach  analyses  can  be  performed  under  both  restrained  and  un- 
restrained conditions.  With  the  shoulder  harness  locked,  the 
model  will  check  for  maximum  reach  without  straining  against 
the  harness  and  for  maximum  reach  while  straining  against  the 
harness.  The  shoulder  harness  is  unlocked  for  the  maximum  reach 
check  in  the  unrestrained  condition. 
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The  CAR  Model  operates  on  the  CDC  6600  computer  under  the 
KRONOS  2.1  Time  Sharing  System.  The  model  is  designed  for 
interactive  use.  It  coaches  and  prompts  the  user  in  the  use 
of  the  model . 

CAR  consists  of  two  stand  alone  programs:  (a)  a Monte  Carlo 

Simulation  Module  which  is  used  to  convert  anthropometric  data 
into  link  lengths  for  a man  model,  and  (b)  a Crewstation  Analysis 
Module  which  is  used  to  evaluate  candidate  crewstation  con- 
figurations. 

Use/Val idi ty:  The  CAR  man-model  objective  is  to  develop  an  expedient,  efficient, 

and  easily  improvable  method  of  assessing  the  realistic  reach 
capability  of  United  States  Naval  Hying  personnel  in  any  given 
upright  seated  crew  workstation. 

An  analysis  of  the  assumed  hypothetical  "constant  percentile" 
aviators  versus  the  actual  aviator  population  indicated  the 
need  to  design  and  evaluate  crewstations  according  to  the  acrual 
measurements  of  the  intended  flying  personnel.  The  CAR  evalua- 
tive method  is  sufficiently  raoid  to  minimize  expenditures  and 
is  convenient  enough  to  ensure  that  the  HFE  analyst  will  employ 
the  technique. 

The  technique  is  better  used  for  system  design  than  test  and 
evaluation.  The  CAR  man-model  makes  use  of  several  preceding 
man-model  efforts  in  that: 

a)  A simplified  link  man-model  is  used.  Such  a concept  proved 
highly  amenable  to  efficient  computer  programming  techniques. 
Just  as  important,  the  concept  also  proves  to  be  a valid 
representation  of  the  human  body  when  one  desires  to  synthesize 
seated  operator  reach  capability. 

b)  The  model  has  joint  angular  limits  modeling  those  of  the 
seated  operator. 

c)  The  reach  zone  concepts  of  MIL-STD-1333  are  retained. 
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The  interactive  use  of  the  nwdel  is  enhanced  by  the  program 
design  which  coaches  and  prompts  the  user  in  the  details  of 
the  CAR  program  procedures.  This  feature  significantly 
reduces  user  training  requirements.  Persons  with  little  or 
no  programming  experience  may  use  the  model. 

CAR  has  been  successfully  used  on  the  F-18  (reach  problem) 
and  the  A-7  aircraft. 

One  disadvantage  associated  with  the  CAR  model  technique  is 
that  it  is  not  a dynamic  model.  Only  static  problems  may  be 
evaluated.  CAR  does  not  recommend  possible  solutions,  it 
only  answers  specific  problems.  It  cannot  optimize  a solution 
to  a given  problem.  Several  less  significant  anthropometric 
parameters  and  relationships  are  not  included  in  the  model. 

If  CAR  is  not  available  for  use,  CGE  or  engineering  drawing 
geometric  analysis  may  be  used  in  its  place.  Mockups  or  proto- 
type hardware  in  combination  with  selected  extreme  percentile 
subjects  may  be  effectively  used  in  place  of  this  technique. 


References : Edwards,  R.  E. , et  al . , Crewstation  Assessment  of  Reach,  User's 

Manual , D180-19321-1  , Boeing  Aerospace  Company,  (NASC  340F), 
April  1976. 


This  work  was  developed  for  the  Naval  Air  Systems  Command  and 
monitored  by  the  Naval  Air  Development  Center. 

A comparison  of  this  technique  with  others,  including  CGE, 
may  be  found  in  Appendix  D. 
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5.6.3  Computer  Accommodated  Percentage  Evaluation  (CAPE) 

Description:  CAPE  is  a Monte  Carlo  computer  model  for  generating  representa- 

tive pilot  anthropometric  features  (including  links)  and  comparing 
these  date  with  an  adjustable  work-space  model  so  that  the  popu- 
lation accommodated  by  the  workspace  can  be  estimated  and  maximized. 
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The  computerized  accommodated  percentage  evaluation  (CAPE) 
model  has  two  options:  exclusion  demonstration  and  cockpit 

analysis.  Each  option,  and  its  underlying  model  with 
components,  is  described  in  summary  form  below.  More  de- 
tailed descriptions  of  model  options,  their  components  and 
the  total  CAPE  model  are  contained  in  the  reference. 

An  exclusion  demonstration  determines  what  percentage  of  a 
potential  population  is  excluded  from  a workspace  design  with 
respect  to  each  anthropometric  feature  entered  into  the  program. 
This  CAPE  option  may  be  considered  to  be  composed  of  two  compon- 
ents, an  exclusion  limits  component  and  a Monte  Carlo  sample 
generator. 

The  Exclusion  Limits  Component  provides  for  the  entry,  storage 
and  utilization  of  user-provided  standard  score  limits  of 
anthropometric  variables  required  for  exclusion  studies.  For 
each  variable  involved  in  an  inclusion  demonstration  analysis, 
high  cutoff  and  low  cutoff  values  must  be  input  by  the  user. 

This  component  of  the  analysis  provides  for  the  sequential 
testing,  element  by  element,  of  Monte  Carlo-generated  standard 
score  vectors  to  determine  if  the  vectors  are  within  the  limits 
set  by  the  high  and  low  standard  score  boundaries  (populations 
of  standard  scores  have  means  of  zero  and  standard  deviations 
of  one.)  Rejection  of  any  component  test  is  defined  as  non- 
accommodation of  that  (sample  subject)  feature  vector. 

The  Monte  Carlo  Sample  Generator  Component  generates  quasi - 
random  vectors  of  standard  scores  that  match  a user-provided 
correlation  or  correlation  square-root  matrix.  It  is  based  on  a 
method  which  generates  standard  score  feature  vectors  with  a 
given  correlation  matrix.  Conformable  vectors  of  quasi-random 
normal  variants  generated  by  a subroutine  are  premultiplied 
by  the  square  root  of  the  desired  correlation  matrix  to  produce 
a quasi-random  standard  score  vector.  This  vector  can  be  viewed 
as  a sample  subject  feature  vector  whose  elements  have  been  con- 
verted into  standard  scores. 
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The  cockpit  analysis  determines  the  percentage  of  a population 
that  will  be  excluded  from  a cockpit  design  based  on  the 
geometric  parameters  of  the  workspace.  The  cockpit  analysis 
option  of  the  CAPE  program  can  be  thought  of  as  being  composed 
of  four  components:  a)  a pilot  link  system  component,  b)  a 

sample  pilot  generator  component,  c)  a component  characterizing 
a seat-cockpit  layout,  and  d)  a cockpit  testing  component. 

The  pilot  link  system  is  an  abstraction  of  a pilot's  anthropometry. 
The  sample  pilot  generator  provides  sets  of  pilot  link  values 
suitable  for  inout  in  the  pilot  link  model.  When  the  compat- 
ibility of  a geometry  is  desired,  these  values  are  entered  as 
the  mean,  and  the  generator  provides  only  this  one  sample. 

However, when  a pilot  accommodation  analysis  is  desired,  this 
generator  provides  quasi-random  samples  from  a multi-variate 
normal  population. 

The  program  uses  batch  FORTRAN  with  a SUPER  FORTRAN  option. 

Use/Validity:  Although  CAPE  has  been  designed  for  use  in  cockpit  analysis, 

design  and  evaluation,  it  may  be  used  to  maximize  the  population 
of  personnel  to  be  accommodated  in  any  workspace. 

Aircraft  cockpits  and  many  other  workspaces  traditionally  have 
been  designed  without  knowledge  of  the  proportion  of  the  user 
population  that  is  accommodated  with  safety  and  full  capability. 

In  aircraft  cockpit  design,  for  example,  designers  have  been 
directed  to  develop  cockpits  that  accommodated  5th  through  95th 
percentile  operators.  However,  crew  systems  designers  are 
designing  for  the  5th  through  95th  percentile  population  only  one 
dimension  at  a time.  The  combination  of  all  the  necessary  dimen- 
sions that  make  up  a workspace  design  limit  the  operators  to  a 
much  smaller  actual  range  than  5th  through  95th  percentile.  It 
has  been  shown  that  more  than  52  percent  of  the  1964  population 
of  naval  aviators  would  be  excluded  when  5th  and  95th  percentile 
critical  limits  are  imposed.  When  the  Navy  specified  3rd  and 
98th  percentile  values  are  used,  over  32  percent  would  be  excluded. 
This  problem  has  led  to  the  development  of  CAPE. 
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Reference: 


One  disadvantage  in  using  this  technique  for  workspace  evalua- 
tion is  that  it  requires  special  training  to  use,  both  from 
the  standpoint  of  the  user  and  the  programmer.  The  cost  of 
computer  use  may  or  may  not  be  significant  depending  on  several 
variables.  It  is  interesting  to  note  that  design  criteria 
do  not  exist  that  will  allow  the  design  of  man-machine  inter- 
faces to  accommodate  90  percent  of  the  population.  MIL-STD-1472 
criteria  will  allow  the  accommodation  of  90  percent  of  the  cop- 
ulation for  only  one  dimension  (or  parameter)  at  a time.  It 
would  be  extremely  difficult,  if  not  impossible,  to  develop  these 
anthropometric  criteria  for  all  workspace  design  situations. 

If  this  technique  is  not  used,  specially  selected  subjects  whose 
anthropometric  data  approach  the  percentile  extremes  may  be 
used  to  test  the  workspace  design.  However,  these  subjects 
are  extremely  difficult  to  find  and  they  are  not  a satisfactory 
means  of  determining  just  what  user  population  is  actually  accommmo- 
dated.  Another  method  is  to  measure  all  significant  workstation 
dimensions  that  relate  to  critical  anthropometry  and  compare  them 
to  the  MIL-STD-1472  design  criteria.  The  effect  of  this  would 
be  to  waive  the  presumed  design  requirement  to  provide  for  90 
percent  of  the  user  population. 

Although  CAPE  may  be  used  as  an  evaluation  technique,  it  should 
be  used  prior  to  formal  test  and  evaluation.  It  is  more  oroperly 
considered  a design  technique. 

Bittner,  A.  C.,  Computerized  Accommodated  Percentage  Evaluation 
(CAPE)  Model  for  Cockpit  Analysis  and  Other  Exclusion  Studies , 
TP-75-49/TIP-03,  System  Integration  Division,  Pacific  Missile 
Test  Center,  Point  Mugu,  California,  December  1975. 

Additional  information  on  this  technique  as  compared  with  other 
automatic  simulation  techniques  may  be  found  in  Appendix  D. 
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5.6.4  Field  of  View  Evaluation  Apparatus  (FOVEA) 


Description:  FOVEA  is  a technique  used  to  test  and  evaluate  the  pilot's 

field  of  view  from  the  cockpit  eye  reference  point.  It  is 
designed  to  plot  the  actual  field  of  view  rather  than  the 
field  of  view  as  represented  by  photographs  and  distorted 
reference  grids. 

The  technique  requires  the  procurement  of  certain  expensive 
test  equipment  such  as  a video  camera,  video  recorder,  a small 
computer,  and  plotter  (or  high  quality  graphics  prints).  Re- 
mote maneuvering  equipment  is  also  required  to  move  the  camera 
and  to  provide  an  X-Y  signal  input  to  the  computer.  The  remote 
equipment  consists  of  a monitor  to  view  the  camera  video;  a 
joy  stick,  force  stick  or  trackball  to  control  the  camera  line 
of  sight;  and  the  servos  to  actually  position  the  camera  where 
desired. 

The  test  set-up  requires  the  placement  of  the  video  camera  in 
the  cockpit  mockup  or  prototype  on  the  seat  pan  at  a set  dis- 
tance from  it  which  would  be  equivalent  to  the  location  of  the 
eye  reference  point.  The  camera  is  manipulated  with  the  hand 
controller  to  trace  the  outline  of  the  canopy  or  windscreen 
windows.  The  remote  camera  operator  (HFE  observer)  knows  where 
to  place  the  camera  by  viewing  the  center  spot  on  the  monitor 
unit.  At  the  same  time  that  the  camera  moves,  similar  signals 
from  the  servo  driving  system  are  sent  to  the  computer  which, 
in  turn,  plots  out  the  outline  of  the  pilot's  view  from  the  cock- 
pit or  appropriate  peripheral  plotting  equipment. 

The  plotted  view  from  the  eye  reference  point  may  then  be  compared 
with  the  appropriate  specification  requirement. 

Use/Val idi ty:  FOVEA  is  an  experimental  technique.  The  primary  advantage  in  use 

of  FOVEA  is  that  it  provides  a means  of  measuring  the  vision  envel- 
opes and  angular  locations  of  objects.  It  records  this  data  both 
digitally  and  graphically  and  in  a manner  compatible  with  specifi- 
cation criteria  for  comparison  and  evaluation.  It  is  not  designed 
to  address  the  adequacy  of  visual  fields  or  locations  of  objects 
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in  terms  of  operational  factors  but  merely  to  produce  objectively 
measured  records  of  what  exists  in  a crew  station.  Additional 
disadvantages  associated  with  FOVEA  are  the  cost  of  the  test 
equipment  and  setup. 

There  are  several  alternatives  to  this  technique  that  may  be 
used.  It  is  possible  to  convert  engineering  drawings  into 
extremely  accurate  comparative  vision  envelope  plots.  It  is 
very  easy  to  take  photographs  (with  the  proper  lens)  and  to  com- 
pare these  with  spec  requirements.  Either  linear  or  distorted 
grids  may  be  added  for  purpose  of  comparison.  Utilizing  a sur- 
veyor's transit  and  a hand  held  transit  vision  plots  of  the  pilot's 
visual  envelope  have  been  successfully  accomplished.  Interviews 
and  questionnaires  are  also  good  sources  of  subjective  data  as 
to  adequacy  of  cockpit  field  of  view. 

References:  Weaver,  A.  J.,  Field  of  View  Evaluation  Apparatus,  Survey. 

and  Analysis;  First  Interim  Report,  SY-13R-76,  Naval  Air  Test 
Center,  Patuxent  River,  October  1975. 

Other  techniques  should  also  be  reviewed  for  comparison: 

Terrain  Visibility  Definition 

Interviews/Questionnaires 

Photographs 

A comparison  of  these  techniques  with  FOVEA  may  be  found  in 
Appendix  D. 

5.6.5  Online  Interactive  Simulation 

Description:  Previous  HFE  T&E  technique  sections  have  described  techniques 

which  rely  heavily  on  prototype  hardware  or  mockups.  Also  in- 
cluded in  this  guide  are  several  techniques  which  do  not  use 
either  mockups  or  hardware,  but  are  instead  computer  program 
simulations  of  both  the  operator  and  equipment  in  the  man/ 
machine  interface  (e.g.,  CGE,  CAR,  CAPE).  The  general  technique 
described  in  this  section  pertains  to  the  use  of  real  time 
computer  program  simulations  and  actual  test  participant  operators. 
Like  other  simulations,  online  interactive  programs  are  used  to 
evaluate  and  demonstrate  the  application  of  specific  procedures 
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and  equipment  to  specific  operations.  It  is  often  difficult  to  i 

! make  a sharp  distinction  between  some  computer  simulation  set-ups  i 

I and  functional  mockups.  The  emphasis  in  the  functional  mockup  i 

I 

I is  on  an  accurate  representation  of  spatial  dimensions  and 

\ arrangements. 

! 

The  most  important  requirement  of  an  online  interactive  simulation 
is  that  it  be  an  accurate  representation  of  some  portion  of  the 
I proposed  system.  Critical  variables  in  the  proposed  system  should 

be  properly  duplicated  in  the  simulation.  In  some  cases,  simu- 
lators must  actually  provide  del iberate  distortions  of  certain 
parameters  in  order  to  yield  operator  responses  that  will  be  valid 
for  the  real  world.  The  use  of  distortions  is  risky  but  often 
necessary  to  compensate  for  some  parameter  that  cannot  be  provided 
for  properly. 

Online  interactive  simulation  presumes  the  use  of  a sophisticated 
computer  and  software.  Test  participant  consoles  must  also  be 
provided  in  a manner  similar  to  the  system  consoles  being  simulated. 

The  preparation  of  test  participant  operator  procedures  is  a first 
step  toward  the  complex  job  of  constructing  the  real  time  inter- 
I active  software.  Online  operation  reauires  the  construction  of 

numerous  operator  commands  in  response  to  numerous  displays  and 
display  formats.  Operator  and  system  performance  outputs  must  j 

also  be  provided  for  in  terms  of  lists  and  time  plots  of  events 
versus  actions,  errors,  and  reaction  times. 

Use/Val idi ty : The  reason  for  using  online  simulation  is  because  of  the  ability 

to  find  out  what  might  occur:  to  manipulate,  to  study,  and  to 

measure  the  model  instead  of  the  real  world. 

i 

I 

There  are  several  advantages  to  using  online  simulation  as  compared 

( to  other  methods  of  T&E: 

S 

L 

[ a)  Simulators  are  frequently  cheaper,  faster  and  easier  to 

' construct  than  the  systems  or  prototype  hardware  they 

[ simulate. 
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b)  Simulators  can  be  instrumented  to  collect  data  that 
would  be  difficult  or  impossible  to  obtain  from  real 
systems  and  the  data  may  be  quickly  reduced  to  usable 
form. 

c)  Simulators  are  extremely  useful  as  training  aids. 

d)  Simulators  are  easier  to  manipulate  than  the  systems  they 
reoresent. 

e)  Simulators  may  be  used  to  perform  tasks  that  would  other- 
wise be  hazardous  to  the  test  participants  (e.g.,  crash 
landings). 

f)  Once  the  simulation  program  has  been  provided,  alternative 
procedures  or  tactics  may  be  easily  manipulated. 

g)  A record  of  data  may  be  kept  for  later  playback. 

The  disadvantages  in  the  use  of  online  simulation  as  compared 

with  other  T&E  techniques  are  as  follows: 

a)  Simulation  tends  to  invite  overgeneralization. 

b)  Simulations  may  be  wrong  because  of  incorrect  relationships 
that  have  been  made  to  hold  between  variables,  or  assumed 
constraints  may  be  in  error. 


c)  Simulators  may  add  ingredients  of  their  own  that  will  not 
be  found  in  the  real  world  system. 

d)  Simulators,  in  general,  are  very  expensive. 


The  time  to  use  online  simulation  is  generally  before  the  con- 
struction of  the  hardware  (and  software)  that  it  is  to  simulate. 
If  this  is  not  done,  there  is  little  point  in  the  expenditure 
of  the  time  and  effort  for  the  simulation. 


There  are  essentially  two  alternatives  to  the  use  of  online 
interactive  simulation.  One  simulation  technique  is  the  use  of 
man  model  programs  such  as  the  CGE,  CAR  and  CAPE  models  previously 
mentioned.  The  other  alternative  is  the  use  of  all  the  T&E  tech- 


niques which  utilize  the  direct  or  indirect  data  obtained  from 
thf  actual  prototype  system  hardware. 
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References: 


Van  Cott,  H.  P. , and  Kir.kade,  R.  G. , Human  Engineering  Guide 
to  Equipment  Design,  Joint  Army-Navy-Air  Force  Steering  Committee, 
AIR,  Washington,  D.  C.,  1972. 

The  OPREDS  (Reference  Section  5.4.6)  is  in  several  ways  a similar 
technique;  however,  OPREDS  is  used  with  actual  hardware.  Addi- 
tional information  on  the  online  interactive  technique  as  compared 
with  other  automatic  techniques  may  be  found  in  Appendix  D. 
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APPENDIX  A:  LIST  OF  PERSONNEL  SURVEYED 


One  of  the  first  tasks  in  the  preparation  of  this  guide  was  to  visit  var- 
ious Navy  labs  and  centers.  Knowledgeable  personnel  were  questioned  in 
regard  to  their  HFE  T&E  experience,  problems,  and  needs.  The  Appendix  D 
T&E  worksheets  are  based  to  certain  extent  on  the  results  of  this  survey. 


The  following  is  a list  of  those 

NADC  - Warminster 
Dr.  L.  Hitchcock 
LCDR  N.  Lane 
D.  E.  Murry 

TECOM/HEL  - Aberdeen 
J.  Perkins 
J.  Miles 
Dr.  J.  Getty 


contacted  during  the  survey  visits. 

OPTEVFOR  - Norfolk 
T.  P.  Enderwick 

NTEC  - Orlando 
LCDR  J.  Funaro 
Lt.  R.  E.  Perryman 
W.  B.  Boney 
R.  G.  Bird 

OPTEVFOR  - North  Island 


NATC  - Patuxent  River 
A.  Weaver 

Dr.  S.  G.  Schiflett 

R.  M.  Walchli 

NAVAIR  - Jefferson  Plaza 
J.  C.  Hemingway 

S.  Miller 

CNO  - Pentagon 
Dr.  R.  Smith 

ONR  - Arlington 
Dr.  M.  Tolcott 
G.  S.  Malecki 


Dr.  R.  Rhea 
LCDR  W.  F.  Moroney 
Lt.  W.  R.  Helm 

NELC  - San  Diego 
R.  Coburn 

LCDR  W.  F.  Moroney 
Lt.  W.  Helm 

NPRDC  - San  Diego 
Dr.  D.  Meister 
R.  Harris 

LCDR  W.  F.  Moroney 
Lt.  W.  Helm 
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AFHRL 

AWACS 

BOEMAN 

CAPE 

CAR 

CGE 

CIC 

CPS 

CTE 

DA 

DACOLS 

DCS 

DEP 

DoD 

DRS 

DSARC 

DT&E 

EEG 

FLAG 

FOT&E 

FDI 

FOVEA 

EKG 

GPTP 

GSR 

HE 

HEDGE 

HF 

HFE 

HFTEMAN 

HPR 

ILS 

lOT&E 

JANA I R 

LED 


APPENDIX  B:  DEFINITIONS/ACRONYMS 

Air  Force  Human  Resources  Laboratory 

Airborne  Warning  and  Control  System 

Computerized  Math  Man-Model 

Computer  Accommodated  Percentage  Evaluation 

Crewstation  Assessment  of  Reach 

Cockpit  Geometry  Evaluation 

Combat  Information  Center 

Computer  Program  System 

Contractor  Technical  Evaluation 

Developing  Agency 

Data  Collection  System 

Data  Collection  System 

Design  Eye  Point 

Department  of  Defense 

Data  Reduction  System 

r Defense  System  Acquisition  Review  Council 
Development  Test  and  Evaluation 
El ectroencenhol ograph 
Feedback  Loop  Action  Generation 
Follow-on  Operational  Test  and  Evaluation 
Functional  Description  Inventory 
Field  of  View  Evaluation  Apparatus 
Electrocardiograph 
General  Purpose  Test  Planning 
Galvanic  Skin  Response 
Human  Engineering 

Human  Engineering  Data  Guide  for  Evaluation 

Human  Factors 

Human  Factors  Engineering 

Human  Factors  Test  and  Evaluation  Manual 

Human  Performance  Reliability 

Integrated  Logistics  Support 

Initial  Operational  Test  and  Evaluation 

Joint  Army  Navy  Aircraft  Instrumentation  Research 

Light  Emitting  Diodes 
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APPENDIX  B:  DEFINITIONS/ACRONYMS  (cont.) 


NAVMAT 

Chief  of  Navy  Material 

NTDS 

Naval  Tactical  Data  System 

NTE 

Navy  Technical  Evaluation 

OPEVAL 

Operational  Evaluation 

OPNAV 

Chief  of  Naval  Operations 

OP REDS 

Operational  Performance  Recording  and  Evaluation 
Data  System 

OT&E 

Operational  Test  and  Evaluation 

PAARS 

Personnel  Activity  Analysis  Radio  System 

PAT&E 

Production  Acceptance  Test  and  Evaluation 

PSTE 

Personnel  Subsystem  Test  and  Evaluation 

RFP 

Request  for  Proposal 

RID 

Remote  Input  Devices 

SOW 

Statement  of  Work 

SECNAV 

Secretary  of  Navy 

T&E 

Test  and  Evaluation 

TECHEVAL 

Technical  Evaluation 

TEMP 

Test  and  Evaluation  Master  Plan 
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APPENDIX  C:  T&E  SECTIONS  OF  MIL-H-46855 


3.2.4  Human  Engineering  in  Test  and  Evaluation  - The  contractor  shall 
establish  and  conduct  a test  and  evaluation  program  to:  (1)  assure  ful- 
fillment of  applicable  requirements  herein;  (2)  demonstrate  conformance 
of  system,  equipment  and  facility  design  to  human  engineering  design  cri- 
teria; (3)  confirm  compliance  with  performance  reqirements  where  man  is 
a performance  determinant;  (4)  secure  quantitative  measures  of  system 
performance  which  are  a function  of  man-machine  interaction;  and  (5)  de- 
termine whether  undesirable  design  or  procedural  features  have  been  in- 
troduced. (The  fact  that  these  functions  may  occur  at  various  stages  in 
system  or  equipment  development  shall  not  preclude  final  human  engineer- 
ing verification  of  the  complete  system.  Both  operator  and  maintenance 
tasks  shall  be  performed  as  described  in  approved  test  plans  during  the 
final  system  test.) 

3. 2.4.1  Planning  - Human  engineering  testing  shall  be  incorporated  into 
the  test  and  evaluation  program  and  shall  be  integrated  into  engineering 
design  tests,  contractor  demonstrations,  R&E  acceptance  tests  and  other 
major  development  tests.  Compliance  with  human  engineering  requirements 
shall  be  tested  as  early  as  possible.  Human  engineering  findings  from 
early  testing  shall  be  used  in  planning  and  conducting  later  tests. 

3. 2. 4. 2 Implementation  - The  human  engineering  test  and  evaluation  program, 
contained  in  approved  test  plans,  shall  be  implemented  by  the  contractor. 
Test  documentation  (e.g.,  checklists,  data  sheets,  questionnaires,  sche- 
dules, operating  procedures,  test  procedures)  shall  be  available  at  the 
test  site.  Human  engineering  portions  of  all  tests  shall  include,  where 
applicable,  the  following: 

a.  A simulation  (or  actual  conduct  where  possible)  of  mission 
or  work  cycle. 

b.  Tests  in  which  human  participation  is  critical  with  respect 


to  speed,  accuracy,  reliability  or  cost. 


c.  A representati ve  sample  of  non-critical  scheduled  and  un- 
scheduled maintenance  tasks. 

d.  Proposed  job  aids. 

e.  Utilization  of  personnel  who  are  representative  of  the  range 
of  the  intended  military  user  population  in  terms  of  skills, 
size  and  strength  and  wearing  suitable  military  garments  and 
equipment  which  are  appropriate  to  the  tasks,  and  approved 
by  the  procuring  activity. 

f.  Collection  of  task  performance  data. 

g.  Identification  of  discrepancies  between  required  and  obtained 
task  performance. 

h.  Criteria  for  the  acceptable  performance  of  the  test. 

3. 2. 4. 3 Failure  Analysis  - All  failures  occuring  during,  or  as  a result  of, 
test  and  evaluation  shall  be  subjected  to  a human  engineering  review  to 
differentiate  between  failures  due  to  equipment  alone,  man-equipment  incom- 
patibilities and  those  due  to  human  error.  The  procuring  activity  shall  be 
notified  of  design  deficiencies  which  contribute  to  human  error. 


C2 


I 

r 


APPENDIX  D:  HFE  T&E  TECHNIQUE  WORKSHEETS 

One  of  the  first  tasks  performed  during  the  development  of  this  guide  was 
the  survey  for  all  of  the  various  HFE  T&E  techniques  and  tools.  As  each 
new  technique  was  "discovered",  it  was  listed  along  with  all  of  its  char- 
acteristics. The  worksheets  in  this  appendix  are  the  result  of  that  ori- 
ginal effort.  Although  much  of  the  data  included  in  the  worksheets  is  also 
discussed  in  the  Section  5.0  narrative,  the  worksheets  are  included  here 
to  assist  the  user  of  this  guide  in  making  comparisons  between  each  of  the 
techniques  and  to  choose  the  appropriate  one(s)  for  use. 

The  following  is  an  explanation  of  the  worksheet  form:  Under  the  heading 
of  "T&E  Technique",  each  of  the  techniques  to  be  evaluated  are  listed. 

Across  the  top  are  a number  of  selection  evaluation  characteristics.  The 
purpose  of  these  characteristics  columns  is  to  make  evaluative  conments  as 
a part  of  a selection  pr  cess.  That  process  being  to  choose  the  most  use- 
ful T&E  techniques  for  a given  program.  Some  techniques  are  obviously  better 
than  others  for  certain  types  of  programs,  program  stages,  or  T&E  efforts. 
The  following  describes  in  detail  what  is  meant  by  each  of  the  evaluation 
characteristics. 

a)  USAGE: 

Specific  Programs  Used  On  - A brief  list  of  each  of  the  programs 
on  which  this  particular  T&E  technique  has  been  used. 

Program  Stage  Applicability  - The  stage  or  phase  of  a program 
that  is  best  suited  to  the  use  of  this  technique:  future  programs 
such  as  the  VPX,  evolving  programs  such  as  the  F-18,  or  existing 
programs  such  as  the  P-3. 

b)  INHERENT  FEATURES: 

Advantages  - That  which  is  particularly  useful  about  this  technique 
as  compared  to  other  HFE  T&E  techniques. 

Disadvantages  - The  drawbacks  or  limitations  of  this  technique  as 
compared  with  others. 

Implications  If  Not  Used  - Significant  consequences  of  what  T&E 
data  might  "fall  through  the  crack"  if  this  technique  is  not  used. 
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c)  BEST  USE: 


Data  Category  - The  category  of  data  (qualitative  and/or  quantitative) 
that  this  technique  would  tend  to  provide. 

Technique  Category  - The  basic  category  (testing  and/or  evaluation)  for 
which  this  technique  would  be  used.  For  example,  some  techniques  are 
useful  only  for  gathering  data  but  not  for  evaluation  of  that  data  until 
an  additional  process  is  performed  to  make  the  data  understandable. 

System  Use  - The  use  of  this  technique  for  all  or  only  a portion  of  the 
total  system  being  evaluated.  For  example,  certain  techniques  may  be 
useful  for  examining  the  HFE  aspects  of  a component,  but  of  much  less 
use  for  a subsystem  evaluation,  and  not  of  any  value  for  a complete  sys- 
tem evaluation. 

Task  Breadth  - The  category  which  indicates  the  relative  quantity  of 
different  tasks  that  may  be  simultaneously  tested  by  using  this  tech- 
nique. 

d)  RELATIVE  PERFORMANCE: 

Relative  Time  to  Perform  - The  time  category  (short,  medium,  long)  that 
best  describes  the  time  to  perform  this  technique  for  a given  test,  as 
compared  to  other  techniques. 

Relative  Complexity  - The  category  of  relative  complexity  (simple,  aver- 
age, complex)  that  best  describes  this  technique  as  compared  to  other 
analysis  techniques. 

Personnel  Required  - Estimates  as  to  both  the  average  total  manhours 
required  per  test  (excluding  both  pre-  and  post-test  time)  and  the  num- 
ber of  HFE  TSE  observers  or  evaluators. 

Relative  Cost  - The  category  that  best  describes  the  relative  cost  (low, 
medium,  high)  of  this  technique  as  compared  to  other  techniques. 

e)  PROGRAM  INTERFACE  REQUIREMENTS: 

Input  Parameters  - The  sort  of  information  that  is  required  prior  to 
initiating  the  use  of  this  technique  (e.g.,  anticipated  human  performance 
in  terms  of  time  required,  based  on  analysis,  or  previous  test  data). 
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Imnediate  Outputs  or  Products  - The  data  that  is  obtainable  from  the 
use  of  this  technique  (e.g.,  operator  errors,  design  criteria  com- 
pliance, need  for  special  tools)  and  what  form  in  which  the  data  should 
be  compiled  (e.g.,  report  format)  and  to  whom  the  data  should  go  (Note:  See 
Reference  10  GPTP  Guide  standardized  format). 

Coordination  Required  - Other  organizations  or  support  that  is  required 
to  perform  the  test  (e.g.,  special  arrangements  as  to  approval  or  secur- 
ity access). 

Remarks  - Any  other  pertinent  comments  in  regard  to  the  use  of  the  tech- 
nique (e.g.,  special  facilities  or  equipment  required  or  special  TSE 
personnel  qualifications). 
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